While it's true that smaller populations have greater growth in percentage points, the absolute growth amounts are higher with high populations. For example; Let's say we have two populations, homeworld of 700m with annual growth rate of 2%, and a colony of 10m with annual growth of 10% per year (which are numbers I've seen in the game). With 2% growth, the population doubles in size in about 35 years, and with 10% doubling time is about 7 years. So the colony doubles five times in the time that homeworld populations doubles just once. So we get the following (assuming that growth remains constant):
Homeworld: 700m*2=1400m => 700m increase in population over 35 years
Colony: 10*2*2*2*2*2 = 320m => 310m increase in population over 35 years
So the absolute growth is over double in homeworld, despite the higher percentage of growth in the colony.
But back to the question. Like Whitecold said, just start the colonies with good leaders and the populations will grow surprisingly fast. And, if there are suitable and agreeable aliens, you could get some labour from there.