I've had the same problems as the OP re. the Swarm ... to such an extent that I typed up the linked report/suggestion for Steve.
http://aurora2.pentarch.org/index.php?topic=10640.msg152683#msg152683. Follow-up posts in that thread, each with new player reports, may be of service to Steve.
Here's what little I know about the Swarm, or merely guess about it. Everything's in spoilers, despite this thread now being in the spoiler forum.
Independent Swarms appear in random star systems, as they are generated (or if you SP place them manually). I wonder if they are preferentially placed in resource-rich systems. There appears to be no check on how distant these systems are to your homeworld, or on the size and strength of your empire, or on whether and how many swarms already exist in the universe. The failure to check for this sort of thing is a general issue with Aurora game design.
A Swarm can travel between stars using jump points (!), which means it can be far more dangerous than the Precursors. Its fleet consists of vessels that search for jump points, that scout for new targets, that serve as major fleet combatants, lots of small fast attack craft, and that harvest wrecks (and perhaps other rescources?) to generate more members of the swarm. Their technology level starts at (I think) about ion or magneto-plasma, and can (slowly) increase over time.
The Swarm means death if you cannot beat it off. If it can, it will wipe out your fleet, your shipyards, and endlessly bombard your colonies to dust. If your empire is facing doom, consider editing the database to remove the entire swarm.
Please report any errors or omissions in the above!
If you start the game with the swarm active, then one of two things will happen:
1. The swarm will find and attack you early. You will either already have a fleet in place that can kill several tens of thousands of tons-worth of ion or magneto-plasma ships ... or you will die.
2. The swarm will either never appear, or appear far away or late, having little or no effect upon the game.
Without painstakingly using the database to determine if systems have had a swarm generated in them, you won't know which of these will happen until they start appearing ... or never do.
I therefore recommend that you turn the swarm off, and - when you think you're ready for a minor, but mobile opponent - manually place a swarm in a suitable system.