NPR don't pay any attention to you when deciding what to do (other than flying fleets of ships to your world to check you out). This isn't nefariousness on their part - they just aren't that bright In particular (as was said), marking a system as yours (or any other empire's) on the galaxy map doesn't affect gameplay at all - it's just a visual decoration.
It doesn't really surprise me that marking a system as yours on the map doesn't affect NPR's. It'd be too easy abuse the system to prevent an NPR from laying their own claim.
If the NPR's completely ignore the player, that's not really a matter of them being dumb. That's a design oversight. It should not be difficult at all to change the program so that an NPR isn't allowed to colonize a system the player already owns a colony and/or outpost in - without first having a state of war, at least. A simple program flag (a binary yes or no) for each system might suffice.
If an NPR detects one of your ships in a system where it has a colony and you don't have the appropriate treaty level with it, then it gets upset with you (diplomacy points decrease). If that happens enough, they'll eventually start shooting at you. Note that the points on the diplomacy screen represent how your population feels about the aliens, NOT how the aliens feel about you.
That seems
quite unfair. So, if I understand correctly, having any of your ships travel their systems - even just passing through to other systems or doing a cursory examination of the system - is enough to start upsetting them - possibly to the point of having them declare war on you.
Even more disturbing, this suggests that they could decide to colonize a planet in one of your colony systems (perhaps even your home system) and
then get pissed at
you for all the ships traversing
your own system?
That's just poor design, IMO...