First off, I try to drop a colony (and only one) in every system with a decent planet. These colonies might not do much, but for logistics reasons I like having at least small colonies in as many systems as possible.
Each populated colony gets, at a minimum, 5 deep space tracking stations and a ground unit with both troops and STO weapons. Ground troops were already covered, but they're both cheap and don't require dedicated maintenance - they cost a little bit of wealth to maintain each year, but wealth is plentiful and renewable, and most importantly they don't cost minerals (which will eventually run out) or require special facilities for maintenance (so I can just drop ground troops on a planet and forget them unless I want to upgrade with new tech). They're also enough to keep the population from protesting about the lack of protection.
Then every few systems I'll have a "Navy Base" system that has maintenance facilities, Naval headquarters, and some ordinance factories that support at least a few warships in orbit. In my current game I'm just using my standard warships, though in the past I've experimented with "System Defense Ships" that are basically warships with the deployment, fuel, and engineering cut down to only last a few weeks. Fighters and orbital bases are both options but not ones I use, since I only heavily defend the bases and want the ships there not to be limited to the system/planet.
The way I figure it is if my jump points go like so:
Earth <> Colony <> Colony <> Navy Base <> Colony
Then each colony always has warships a minimum of one jump away.
My rationalization is that if a colony gets attacked by a small force, the STO weapons and troops can hopefully hold out, at least until reinforcements arrive. And if it gets attacked by a large force then a few OWPs or fighters probably wouldn't make a difference anyways.
Of course, if I know there's a possible threat out there, like another race that's been probing my territory, the colonies on the border get heavily reinforced. If I used OWPs and planetary fighters, this is probably where they'd come in. The ground forces alternating with navy bases is just my standard for areas I don't know are under specific threat, because you never know when a dormant jump point or new contact will drop enemies on your lap.
Another possibility is minefields, but I understand they're buggy in the current version.