Micrometeorites, other debris,the ship constantly having pressure pushing outward, wear on parts, stress from other sources (gravity, magnetic, etc), and a bunch of other stuff.
Small impacts are not even remotely going to cause a problem. These ships have armor capable of withstanding nearby nuclear weapons. Micrometeorites aren't going to do more than scratch the paint.
Pressurization stress isn't a problem. It's a constant load, and constant loads within design tolerances are basically negligible from a lifespan point of view for anything. What kills airplanes is fatigue from cycling. Ships with 30-ft draft have similar pressure loads on their keel plates, and they don't have any issues at all from this, even when they're 70 years old.
Wear on parts won't happen because the ship is basically abandoned. There will be some degradation, but it will be relatively slow compared to what would happen if you parked a ship without maintenance.
The other sources of stress are totally negligible. Gravity and magnetic effects are of interest to the attitude control people, not to structures, unless you're doing something really wacky. (This isn't really wacky.) And I can't think of anything other than radiation which falls into the 'bunch of other stuff', and that's probably a big component to the slow rate I see happening.
Space is not a nice environment, but spacecraft are designed to deal with it passively, as opposed to ships, which generally take quite a bit of work to keep in good shape. Turning the spacecraft off is only going to reduce the amount of work needed.