Are passives the biggest offenders due to there not really being a cap on detection range for them?
It's really the fact that each race runs its own sensor check routine and tries to detect every object of every other race. So assuming every race has 10 ships and each one tries to detect each ship of every other race, 2 races will be 100 checks. 10 races will be 10,000 checks. Its actually more efficient than that in reality as sensors are tracked on a per fleet basis and as soon as one fleet detects something the other fleets don't bother to check it. However, more races means more fleets and each race tries to detect all ships of all races.
The early game of a large multi-earth start can be very slow in performance terms and this is an effort to get over that initial hurdle.
Steve