There is no code related to chance to hit being affected by dust levels on planets. Given the very high chance to hit (for ground bombardment) of the early shots, it may be that the earlier shots targeted units with lower levels of fortification.
Ok. So we have some randomness involved here.
So how do we deal with this? We'd love to capture their colonies intact or at the very least only partially damaged, but with the tech level of the opponent and the sheer amount of STO units I don't see a way to get enough armies down there without massive losses on the way in.
As things stand we can simply quarantine the system for a while and have them neutralized as a threat, but as gamers that's pretty unsatisfactory.
When faced with hard to tackle STOs, I have used a few different methods. If the STOs, don't have a range advantage, you don't have a problem so I assume that's not the case.
You can use ships with large shields, then back off out of range and recharge when they drop near zero. Without shields, use a similar approach for armour and have a base or repair ships nearby. It takes a lot longer but you wear them down eventually.
Or if the STOs are in difficult terrain, such as jungle, keep attacking the planet with beam weapons or missiles until the dust forces the temperature down far enough to kill off the Jungle

Or you can build massively armoured troop transports, escort with warships and take your lumps on the way in. If you are going to conquer a home world, its difficult to see how the loss-reward ratio won't be in your favour, even after massive losses.
I've used all the above in different situations. Ultimately, your advantage over STOs is that you can choose the range, you can back off when needed and you have plenty of time. Look on it like a siege.