Apologies for necro'ing an old thread but the first poster did not get an answer to his question on how to stop communication attempts with aliens.
In the "intelligence and foreign relations" window (upper right button on the system map) there is a button called "remove race". This does not remove the race, it only erases your knowledge of it. As soon as you make contact again, you start fresh from your perspective (the aliens may disagree).
This is also how to restart attempts at communication if the previous contact team just gave up: "the contact team believe this race is too
alien and communication can never be established. No further attempts will be made. "
From a roleplay perspective: the previous contact team tried to hide their incompetence by blaming the aliens for being alien, so you kick them out the airlock and demote them to painting spaceships for the rest of their lives, while you bring in a new contact team that hopefully is more competent.