Ok, I have browsed the forum for some more tip... I have now restarted a new campaign. The problem that I had before were clearly too man NPR that were active in the background.
I started a conventional start but modded me into a very early Newtonian age to sort of get my empire running. I more or less advanced my empire about ten to fifteen years from a conventional start position. Sol is pretty much explored and the first Explorer ships is rolling of the production line who will take humanity into the stars and beyond. My tech are now around nuclear pulse engines and about that tech level in the more important areas of technology.
I set it so that I had one NPR from the start but there is now only an 8% chance for any other to spawn so they should not become too common, I limited the number of systems to 300 (plenty enough I guess). When the game do start to slow down I hope that I have such a sprawling empire that there are so much to do between turns anyway that the slowdown will not be such a pain.