Ground Forces can be clicked and dragged from one colony to the other at the same population. I don't remember which window this happens in offhand, but one of the ones where you can see the formations listed under the populations should do it. After that just delete the unused "colony".
I assume the reason this is not automatic is if you have an actual colony and not just a placeholder for unloading ground units, which would happen if multiple races had a colony at the same body and eventually fought a war about it.
If you have ruins on the body they are tied to the body, not a specific colony AFAIK, so they should remain if you delete the other colony. That being said, if there is loot in the other colony it is probably better to shift the ground units as above so they can guard the loot instead of deleting it.