DB size definitely does matter for performance. Worst part is all your saves are in the same DB, so starting a brand new game will perform better the first time than on the 20th time. I didn't notice any real event-related slowdown until ~100 years and three ground invasions, two of which were heavily-defended homeworlds and so generated an absolute mountain of events.
Edit:
My current DB with that game I mentioned above is at ~78mb.