What is Earth's population cap? The laws of physics don't seem to cause one; people don't suddenly become infertile once the population hits X billion. Perhaps there is a point where everyone feels too crowded, but most of the Earth is effectively empty; we're not anywhere near it today. The real limits are set by the fraction of land which can be used for agriculture, the amount of agriculture which can be accomplished without using arable land, and the number of calories you can get for a given set of inputs. Technology has given us such great improvements to the latter that we're actually not using all of the arable land that we could be, and we still have more food than we can eat. I think this is a situation where the trans-newtonian technologies that the game introduces have pushed the limits so high as to be entirely unknown.
The rest of your suggestion is asking for more simulation here and less abstraction, so I don't think a cap of any kind would be the right way to satisfy you. I think terraforming could be more interesting if it took into account more factors about the planet though, such as the hydrology.
Yep. Rather than a hard cap, having to take care of the hydrology (too many or not enough water = problem) and ecosystem (do I need to import species from earth? Is the local ecosystem compatible for humans, if life already existed), etc... Would be more interesting.
Rather than having a fixed population size depending on these factors, you could have a reduction to efficiency for the planet if those points are not fulfilled. Or a need for infrastructures to cover for producing food in an unsuitable environment (right atmosphere but no water, or no specie to cultivate, interaction between native ecosystem and terran crops...).
As a whole, I think terraforming could use more detailing for what happens once temperature and atmosphere concerns are addressed. For example, atmosphere without an ecosystem could degrade with oxygen turning to CO2. Pollution could also be an issue worth adding, requiring you to use terraforming installations and workers to remove dangerous gases from your otherwise breathable atmosphere.
Another thing worth thinking about would be alternative terraforming methods, like using ice asteroids to create an atmosphere and hydrosphere quicker than by adding gases manually, or introducing genetically altered flora converting gases (CO2 to O2 ...). Right now, terraforming seems to be too linear and straight forward. If a planet is colonizable, it is usually terraformable, and you always proceed the same way to do so. It reduces the thinking needed when choosing settlement targets since most worlds with the correct gravity can be terraformed to your specie's liking. Having to concern yourself with the existence of a magnetic field, the day length, seasonal variation strength, tidal lock, volcanic activity polluting the atmosphere, etc, could make it much more interesting and reflection inducing.