I don't think anyone would, short of cataclysmic upheaval or a radically alien psychology among the Venusians or the Martians. In each case, the majority of the population is based on the planet of their species' birth, and even if most of the population moved out to colonies, I would think the species would keep a caretaker population and a strong military on their homeworld rather then let another species claim it.