For a ship of a fixed size with a fixed amount of HS devoted to defence, shields and armor compete for the same hull space. This is not a conditional, they do at all times. They both perform the same function, which is to say keeping your ship alive.
Any given HS of Armor's contribution to the effective number of hits we can take without dying or being crippled is a function of ship size (bigger is generally worse) and the the penetration of the enemy hits (so their damage profile) with shallower damage profiles being better and deeper damage profiles being worse. And of course a technology based armor efficiency constant.
Armour absorption = f(ShipSize,Penetration)*Tech(Armor)
Any given HS of Shield Generator's contribution to the effective number of hits we can take without dying or being crippled is a function of a combination of time between attacks and regeneration speed, but restricted to a certain minimum (shields have 0 time to regenerate) and a certain maximum (shields regenerate to full in between volleys), both based on a technology based efficiency variable.
Shield absorption = g(TimeBetweenAttacks,RegenerationSpeed)*Tech(Shield) with g() being constrained as mentioned above.
(Shipsize is fixed under our assumptions. RegenerationSpeed is also related to technology and can therefore be assumed to be fixed for a given technology level. Penetration is under our enemy's control, but also depends on combat distance, etc. Regardless, we can figure out a general damage profile we want the maximum defence against (2-3 penetration earlier on, with more later) and we know attacks will likely hit one of the two extremes for g(): too fast to regenerate (Beams/AMMs) or slow enough to allow full regen (ASMs).)
Note that neither value depends on either how much shield or armor you already have. That means that if you have 2 HS available for defence, and your calculations reveal that for the 1st HS, Armor is a better option, then for the 2nd HS armor will STILL be a better option since none of the factors changed. If anything, shield synergizes slightly with itself.
In summary, it doesnt actually matter which one of these win in any given situation. The point is that ONE OF THEM WINS. And either Shield Absorption is always better, or Armor Absorption is always better and it only depends on your tech level.
The reason "Leaky shield" overcomes this, is it makes the value of armor related to the number of shield layers you have (penetration becomes penetration - shield layers) and shield value related to the amount of armor you have (your shield only has value if you dont DIE before it is fully depleted). The optimal solution is therefore moved away from the extremes, towards the centre of a combination of shields and armor for defence which, and this is purely my opinion and you are welcome to disagree, I feel leads to more interesting gameplay decisions.