I tend to agree with everyone else that just rescaling the profit/loss for shipping lines won't solve the problem, just push it down the road a bit. That being said, I do think some suggestions here are a bit broader than needed to accomplish the stated objective:
So I am considering changing how shipping lines work, so they don't become so overwhelming, while trying to retain the flavour.
emphasis mine.
In my view (and from reading comments here), CSLs become overwhelming when they have too many ships - this unbalances the economy (due to tax income), trivializes new colony growth, causes rapid (de)population of colonies which disincentivizes use of source/destination automation, and causes performance problems (although this was mitigated by the change to use only larger ships).
Why do CSLs build too many ships? Simply put, because they keep making money and have no other way to use that money - there are no other money sinks for CSLs. This brings us back to:
Another option is replacing dividends with a percentage maintenance payment, which is modified by an admin overhead that increases as the number of ships increases - effectively limiting the rate at which a shipping line can increase in size.
I think this is probably the best starting point - as the saying goes, the bureaucracy is expanding to meet the needs of the expanding bureaucracy, so too goes the shipping company.
I would suggest implementing a civilian maintenance fee with superlinear scaling per ship. If N is the number of ships, something like N^(3/2) or maybe even N^2 (maybe this is too much) should work. If the rate of profits depends roughly on N but the rate of maintenance depends on N^(3/2), for instance, then there will eventually be a hard limit on the number of ships a CSL can buy and operate. This also means they will grow more slowly as they come closer to this point.
I would also suggest that CSL income is affected by the wealth generation tech level of their associated race (while the maintenance fee scaling remains the same throughout the game). This would mean the limit on number of ships is initially lower but increases with tech level, which is a rough analogue for the growth of the player empire.
I think this is the simplest and best approach as it is more or less completely under the hood (except for tying to research, which is only a positive change IMO). The essential flavor and interaction with CSLs otherwise remains the same, they just become more manageable. I don't like, for example, ideas about tying CSL growth to number and sizes of populations, I feel like this creates a danger of trying to micromanage populations to manipulate CSL growth (whether to speed up or slow down), which is not a style of gameplay I think fits Aurora.
----
There are a few other suggestions in this thread which I think merit consideration. They do not address the issue raised in the OP, at least IMO, but they are good ideas anyways:
Consider that civilians just use 50% power reduced engines and keep the higher speeds of those ships.
Agree as this makes the minimum engine power tech more attractive and will contribute to limiting CSL growth rates due to higher costs per ship.
--- Perhaps give them a sort of fluff base? Like CMCs, they'll spring up on eligible worlds, but would serve as mock "supply bases".
This would probably be too much work, but I would highly approve of CSLs having to build/expand their own shipyards for example. Being able to pop a 200,000-ton freighter into miraculous existence with no orbital infrastructure is a little silly.
At the very least can we get the oft requested reserve/target min/max settings for population, similar to whats already in place for minerals and installations, so that our source colonies aren't perpetually evacuated if left unattended.
It is oft-requested, but I would like to add my support here. In games with multiple player races, trying to keep track of all source/destination populations across multiple races is tedious at best and impossible at worst, for a single player race maybe it is "more immersive" not to have this automation option but for multiple player race games I think it is essential. As it is, I turn off civilian shipping in multiple player race games to avoid these problems. Implementation could be simple: have a trigger population value set for a source or destination colony, and when it is reached set that colony as stable - no or minimal changes to the civilian AI are needed.
This being said, these suggestions mean potentially significant mechanical changes and a lot more work on the coding side, so while I like them I wouldn't say they are necessary to solve the problem.