Presuming an actual npr not a spoiler. To provoke them all you have to be is detectable , so getting shot is just a matter of being in range and they have warships for that. As I said before there is no distinction between a ship in orbit and a ship at the jump point as far as getting you negative diplomacy points as far as I am aware.
In general a diplo ship does not provoke much but still works better if it is not in their home system but in detection range of another ship or colony. However if you had other ships in the system that may have provoked them.
Usually before an NPR fires on you there will be messages demanding you leave the system, if you do not yet understand the language these will be incomprehensible transmissions noted as such in your log. If there were no transmissions and or your log said the aliens were not responding to attempts to communicate particularly if there is no large population you are aware of then probably spoilers
Ah ok - everything seems to be pointing at your description of spoilers - there is however only one ship above their planet and in the system of many planets they only have one colony
Which leads me to the question I was going to ask next - are there super sneaky ways at getting a peep at the colony to get more info