The gameplay reason is to avoid having very large hierarchies filled with formations with no commanders and relatively junior officers at the top.
An alternative that crossed my mind while asking the question would be to have some sort of "time in grade" requirement for promotion, with a game mechanic that allows (but penalizes) "out of zone" promotions by the player. I didn't mention it because I couldn't think of a way to make it work, but since that's one of your concerns I think I'll throw it out there and see if anything sticks....
The idea is to let players quickly move good commanders up in rank (they are good commanders, after all), but have resistance to that movement from experience/political realities (to avoid the failure mode you're concerned about). A few thoughts:
1) Have an "experience/time in grade" clock, which sets the pace for normal advancement. The idea is that a brand new Captain will have lower skills (read bonuses) than one who's been commanding a major combatant for 2 years. So a newly promoted officer would get penalties to his/her command bonuses that would decay away over time. If an officer got promoted up the chain too quickly, these penalties would compound and destroy the utility of said rapid advancement. This is why navies have time-in-grade requirements for promotion; to avoid too-rapid promotion.
To be clear, the time-in-grade clock would be compared to a "can promote" duration (e.g. 5 years); if the officer has been in grade longer a promotion would be allowed and the clock would reset. There might also be a "auto-promote" duration (e.g. 7 years) which would force promotion for political officers.
2) Allow modifiers to the clock. The big example here is battle experience: every battle where the officer was in command of an element might advance the clock by a fixed time; if the officer was serving in a staff role that time would be lower (I think this is what the medal mechanic is supposed to do in the current system). In the absence of combat, a position in command of a ship or on a fleet staff might also run the clock faster than other positions; "unassigned" might run the clock more slowly.
3) If you go down this road, I actually think you might want two clocks: one for experience and one for time in grade. That way the political bonus could affect how quickly officers are promoted (time in grade clock would run more quickly for high political bonus), without running their clock faster on the experience front.
4) Ageod's American Civil War (I & II) handles this in a cool way, except I don't see how to port the mechanic into Aurora. When generals win battles, they rise or fall on the seniority chart for that rank, and occasionally become promotable. The trick is that you can only promote the most senior general at a rank without paying a political penalty (negative effect on National Morale IIRC which affects productivity). In Aurora terms, if you promote a politically un-connected officer past one with a high political rating, you would pay a price in terms of your empire's efficiency. I can't think of such a morale rating in Aurora, though, to hook this into.
If you can figure out a good penalty, then you could just allow out-of-zone promotions in the current system and not worry about the clock stuff discussed above. Hmmm - maybe a penalty to command (and/or other) ratings that decays away with time, to abstract away inexperience and/or lack of legitimacy in the minds of the crew and/or superiors. If you did this, the decay should only happen when the officer is assigned - that defeats the exploit of rapid-promoting early, then putting the commander on a shelf while the penalty decays away.
Actually, now that I think of it, AACW is trying to do exactly the same abstraction that you are: a small number (~100?) of exceptional commanders in a sea of unnamed officers. They abstract it through their "command points" abstraction, where bigger formations get an overall efficiency hit if they don't have a (named) commander with enough command points to manage the formation. So the highest ranking general at a rank might only have a seniority of 3 (low numbers are higher rank).
5) Not an idea for the above change, but something that just occurred to me: in chain-of-command succession, it seems like named officers always win over unnamed. In other words, if I've got 6 ships in a TG, and only one has a named commander, that commander is in command of the TG even though there's 5 other unnamed ones that might be senior to the named one.
Like I said - I don't see a way to make this (out of zone promotions) work in Aurora, but it seems like a good way to reduce the burden of the "officer pyramid" if you can. To be clear, the problem I think this solves is that the number of officers required to get to a new highest-level rank grows exponentially, so players end up tracking and assigning tons of low-level officers to support the much smaller set of captains and admirals. It would be nice if the highest rank (read organizational skill) available in a fleet was more closely related to experience than to raw numbers.
Another thought: It would be nice if there were a mechanism where named officers could be "noticed" at all levels and have stats rolled at that point (or if the game managed unnamed officers behind the scenes with stats already rolled, but didn't display them to the player until some rank threshold is reached). This probability could then be skewed to let players focus more on the higher ranks, e.g. all admirals would be named, 1/2 the captains, and only 10% of the junior officers. This would also solve the "problem" with named commanders always out-ranking unnamed commanders - if the ships with unnamed commanders had an actual commander assigned and displayed to the player, then one of their commanders might end up senior. (You'd need to set it up so that popping a named commander onto a ship then removing him didn't change the unnamed commander to prevent players churning unnamed commanders to unnaturally get low ranking ones in command of ships.) Even more interesting would be to have the unnamed officer stats actually affect combat, without the player being able to know the modifiers that are being applied.
John