So first off, I used Designer Mode to find this bug (if it is one), however, I haven't made any changes while in DM, just used it to find what the hell is going on.
As for the rest of this, I'll spoiler it, since it involves some of them.
After my latest game had slowed down to 6 hour turns, I decided to use DM to find out why, exactly, that is, because the turns were too long to be combat-related. It turns out it was the Invaders. They were the only ones having any interupt events in those timeframes. It was hostile transit detections by their gate constructors spread all over the galaxy. Quite large numbers, too (some in the 100s). Every 6 hours the game stopped to announce the hostile transit of various amounts of ships, but no contacts were to be seen anywhere. None of the other empires or races, Star Swarm included, were even active in those systems, in fact the Invaders were the only ones who had actual knowledge of those systems at all, none of the others had been there yet. So either I massively overlooked something, or the Invaders have detected "phantom transits". Removing the gate construction fleets from the systems in question got rid of the interupts, tho they did reappear shortly thereafter with new transits in different systems, same problems applied here, nothing to be actually seen.