As far as I know and can tell, the biggest slowdown is drive access.
Other than that - Did you make the mistake I did a while back and start a game with a large population and NPRs turned on? If so, the AI probably has about 1000 civilian ships, slowing you down.