The simplest solution might be the best. Just let players/NPRs mark systems as theirs on the galactic map. (auto mark with colony) Have these transmitted once communications are open.
In the meantime, current mechanics are plenty good enough. You CAN put probes on the outer jump points of your territory. (I suggest doing so anyway) You can get less subtle by using cheap attack fighters, or minefields.
The problem isn't that we really need a new mechanic. The problem is that the AI doesn't understand such concepts as diplomacy or 'get out of my territory'. Perhaps the AI should be made to leave systems claimed by others, depending on some value and current relations? In the strategic intelligence window there's already a list of 'known systems'. Leverage that? Make the AI generally understand warning shots? (with potential to go hilariously wrong?) Or powering up active sensors/shields as a warning? Or just the concept of having a ship stationed at a border?
Come to think of it, these NPRs are, in fact, alien. Perhaps them not understanding such things makes sense.
I suggest that anything like cultural bounds is going to feel FAR too gamey when applied to a game with a human controlling the races.