Arwyn, so much of what you wrote is exactly what I feel.
4e: Yucky technical systems with little to no flavor, confusing r&d rules, upgraded system generation rules,
There are ways to mitigate the "risk" of encountering a hostile NPR early. For starters, you could rule from on high that you encounter no NPRs for 1 game year. Might be a little boring for the people looking for battles, but if you're thinking long term, it might give you a chance to do some exploring and get started on some colonizing. Or you could make it, no NPRs within X transits of your homeworld, if you want to explore and colonize a bit more slowly (in game terms). That is, with a set time, some players might take the opportunity to explore outwards like crazy. Indeed, with a set time, you might almost have to explore like crazy, just to keep up with the jones. So, perhaps a set number of "no NPR" transits from your honesystem might be better. Actually, Ultra (and perhaps earlier versions) set up a 1 transit buffer zone, but for those who are really worried about early nasty NPRs could just make a deeper buffer zone.
As for the randomness, therefore the inherent imbalance, of system generation, you are exactly correct. Like I said earlier, if you want perfect balance, get rid of all NPRs and have only a single template for all star systems. That'd be perfectly balanced. And it'd be perfectly boring. Any process that creates the high degree of randomness and variety as does the Starfire system generation process does, particularly when you add things like anomalies and galactic oddites, is by definition going to be random, inconsistent, and unbalancing.
However, if you're playing a larger game, as Steve and others have described, I think that you could also say that statistically, things will tend to even out in larger Starfire galaxies. Sure, the NEXT star system thru the NEXT warp point may be horribly rich, may have a terribly nasty NPR (or a very friendly NPR), be a deadly nebula, pulsar, neutron star, or black hole. And sure, you could have bad luck at a bad time, but them's the breaks. If you can't deal with it, you're playing the wrong game. People that want a game that 100% skill should be playing something like chess.
I also agree with your comment that the 4e weapons seem so bland. Everything's just bland weapon X of generation Y. There are no "Heterodyne Lasers", no SBM's, no flavor. If anything, 3e could have used more flavor, not less flavor as has been seen in 4e. "Ooooooo, I'm soooooo excited. I just developed the 'e' generation of Lasers. I'm sooooo excited." Spare me.
I'm not the history major that you are, Arwyn, but as a mere layman I've come to the same general conclusions as you. R&D is an effort to create imbalance. There are some evolutionary developments in weaponry, but the way that I see it with my layman's view, that's what happens between the big breakthroughs. People don't stop trying to make things better, of course, but the real goal is to find the next magic bullet, or golden bee-bee, or whatever you want to call the "next" big breakthru that gives you a major advantage over your enemy, even if that advantage is fleeting... just as long as you can use that advantage mercilessly until your enemy manages to duplicate it.
Ah well. I think that I'm nearly done on my Ultra-ized system generator, and perhaps within a week or 2, I'll be ready to start my new solo campaign with my 3rd "Fred" rules.