Maintenance is a rather complicated set of mechanics, in part because documentation is a bit hard to find compared to most other mechanics. However there are a few bits and pieces which can be combined to try and understand the full picture:
- Failure rate is proportional to ship size, specifically 0.02% * ship_HS / engineering_size_fraction along with some multiplier based on the maintenance clock
- DAC% is proportional only to component size, notably it is not in any way related to cost or HTK. Of course armor has zero DAC%.
- Engine HTK is equal to SQRT(engine_HS); I'm not sure offhand if non-integer results are rounded or floored, but this does imply certain HTK break points either way.
- As stated in the OP, component failures are modeled by a 2-damage roll against a component selected from the DAC% table.
- The repair cost of a component failure is simply the component BP cost (paid of course in MSP).
Based on this we can see how to
estimate the maintenance life, although this would be a messy calculation that is intimately dependent on the actual components loaded onto the ship - it isn't enough to just say "the ship is 120k tons with 40k tons of engines", because all of the other components will contribute to the estimated maintenance life.
That being said, we can evaluate the maintenance cost of engines fairly independently at a less abstract level. Consider two engine sections of the same size and therefore total DAC%, let's say that one engine section has 8x size-25 INPE engines (cost 125, HTK 5) and the other has 2x size-100 INPE engines (cost 500, HTK 10) - for right now we will ignore the difference in resulting fuel range. The 8-engine section has DAC% chance to roll a 'hit' on a maintenance failure, and once hit has a 40% chance (2 damage / 5 HTK) to suffer an engine failure requiring 125 MSP to repair; the expected cost is therefore 50 MSP every time the engine section is targeted. On the other hand, the 2-engine section once hit has a 20% chance (2 damage / 10 HTK) to suffer a failure requiring 500 MSP to repair, costing 100 MSP on average when the engine section is targeted.
So the smaller engines are better right? Well, not necessarily. As skoormit suggested, the solution here is in part that we need to look at both engineering spaces and maintenance storage solutions. Assuming that the ship has the same size (say, 600 HS or 30k tons), we can achieve a specific AFR% for a specific number of engineering spaces (say, 24 engineering spaces for 300% nominal AFR). Now consider a balance of fuel and MSP spaces. For the 8-engine section, if we suppose that we need 1/3 as much fuel as engines we have about 66 HS of fuel storage. For the 2-engine section, since fuel efficiency scales with SQRT(engine_HS), we only need half as much fuel to achieve the same range, some 33 HS of fuel. The larger engines do cost twice as much in maintenance requirements, but at the above calculated 100 MSP per engine failure roll compared to 50 MSP for the 8-engine section, just a single 50-ton maintenance storage bay (+400 MSP) will cover the difference and leave us with 32 HS left over that we saved on fuel storage. Of course, that 32 HS of extra components could also increase maintenance requirements over the cost of fuel storage, so the 2-engine ship might look "even more expensive" but it is also performing better with another 32 HS of weapons, shields, or whatever.
Of course if we had only looked at using engineering spaces to get the same AFR%, we would have needed a lot more than 1 HS (though not as much as double, since the other components are about the same), so of course the bigger engines do not look as good in that comparison. But as we have seen it is not the best basis of comparison.
----
It is also worth noting that while 3:1 engine:fuel ratio is theoretically optimal, in practice there are many good reasons to use a different ratio - nearly always higher in terms of engine mass (4:1, 5:1, 10:1, etc.). The main one is fuel conservation, not only to reduce resource use but also to ease logistics as a fleet which uses less fuel is a fleet that is easier to support with tankers or fuel depots at a distance. In practice I think people rarely use 3:1 outside of theorycrafting exercises because of this factor. Usually, it is better to think of 3:1 as a limit rather than a rule. If you use a higher ratio of engine size, the resulting ship design is slightly less efficient but you will conserve fuel better. However, if you use a lower ratio of engine size, say 2:1, then there is no benefit - the design is still inefficient, but you are also consuming excessive fuel. Basically, it is fine to have more engine size versus fuel size than a 3:1 ratio, but almost never is it a good idea to have less than 3:1.
The other thing to consider is that engines with less than 1.0x engine power modifier have an additional cost reduction by that modifier. That is, while an engine with (say) a 1.5x modifier will cost 1.5x as much as an engine with 1.0x modifier of the same size, an engine with (say) 0.5x multiplier costs only 0.25x as much as an engine with 1.0x modifier of the same size. This means that very large engines with sub-1.0x efficiency can become very cost-effective since the repair cost (in MSP) is the same as the build cost (in BP), in addition to the benefits of fuel efficiency. Of course this comes at a performance cost as you will need quite large engines to achieve a desired speed, you will likely be looking at quite large engine:fuel ratios which does take away space you could use for weapons, sensors, etc.
What this means for ship design ultimately is that you need to look at the whole picture - balancing build cost, maintenance life, fuel efficiency, and per-ton performance to get the best ship for your race-wide strategic situation. I will say that usually it is not the best approach to think about optimizing tonnage, usually the actual costs of a ship in build, research, fueling, and maintenance costs are much more important limiting factors. You may be able to take a 18,000-ton warship design and instead create a design with the same capabilities on a 20,000-ton hull which has the same build cost due to using cheaper engines (say 90% efficiency instead of 100%), but uses only 85% as much fuel to achieve the same operating range. The only importance of tonnage is the effect on your shipyards, as you do not want to overbuild your yards and tie up excess population, but usually you can be pretty flexible on tonnage if you are willing to think a bit outside of the box.
For most warships, I think, engines are not a very large portion of the cost.
This is not really correct, I'd say engines make up 30% or 40% of the cost of a warship as a general rule, which is usually if not the largest fraction of component cost in the top 2-3 for most ships. I think the only other component type that can reliably make up a greater fraction of the cost than the engines is a large military jump drive.
This is unlikely to have a clean solution to this problem. As the engineering spaces required to keep a ship at a certain maintenance life depend on not only the size of the ship but also the total cost of the ship.
To quote ancient Jaffa wisdom: indeed.