I shockingly established communications with a warship guarding a jump point (my diplomatic ship hadn't even made it to the system). Does this make my diplomatic ship useless now? When it caught up to an alien ship, they asked it to leave.
Your diplo ship can still communicate with the NPR to try and raise their diplomatic opinion of you. I recommend putting them within contact range in a relatively innocuous system (not their home system or another that is too close).
The aliens have told me to vacate an empty system next to what I am sure is their homeworld. That is fine with me. However, they keep jumping into one of my systems and running around.
Is there any way to tell them to stop? Can I claim a system the way they claimed an empty one?
There is a tab in the Intelligence window where you can set a message to send to the NPR ships in a given system. They may decide to accept or reject, and different messages have more or less dramatic consequences to interplanetary relations if rejected (a suggestion is less provocative than a demand, etc.)
You can set a default message for all newly-encountered NPRs in the Galactic Map, there is an option in one of the tabs to do this.
What are the consequences if I blow these (probably) survey ships up? Is that automatic war or will they get the message and stop coming?
It will be a fight to the death.
When I played years ago, I seem to remember I could scare them out of the system by activating a fire control, targeting them, but not opening fire (they would jump right back out of the system). Am I remembering right?
Nope.