Going back to the optimization question, although I have to admit I try to avoid going into the math on something I do for fun, the full situation probably is helped if you stick a few formulas into the mess.
To determine the chance a missile will hit the target you are looking at a compound equation that is simple:
final_to-hit_chance = ((1-f)^n)*((1-g)^m)*(1-h)*k*l
f is an expression that takes into account the chance the missile is intercepted by long range counter missiles. This is unfortunately complex. It depends a lot on the enemy's technology, sensors, fire control, missile design, rate of fire, etc. It also depends on if they are firing in 1v1 or something else. It also depends on your missile design: Armour, speed, ECM have effects on this.
n is the number of counter missile salvos you will have to evade.
f and n are linked and quite frankly vary from player to player since you are likely fighting different designs and so on.
g is an expression that takes into account all close range point defence fire on a missile from consorts. It is influenced by the enemy defences mainly, but also by your missile parameters.
m is how many attempts are made and I'm simplifying somewhat in that I doubt all ships assisting have a similiar chance to hit. Likely the (1-g)^m should be (1-gm)*(1-gm-1)...(1-g0)
h is the point blank point defence fire probability function of the target
k is a a fuction of missile range, launch range, target velocity, missile velocity, and missile endurance. This says if the missile can hit the target. It will have a value of 1 or 0. You should ignore it during an analysis and simply say I will always launch when I can hit the target for sure.
l is the basic to-hit formula (missile speed/target speed)*agility
Now it should be, I hope obvious, this is a non-tractable problem. It is dependent on circumstance (ex: launch range), target, and launcher. There is unlikely to be a single optimal solution but instead a large area of parameter space where the overall expression is maximum and possibly this is not single valued so you have several possible solutions that give you essentially identical maxima.
If you make even some guesses on what these values are then you can do a simple bit of number crunching to get a extimate on effectiveness:
So if I just toss some numbers in there (from Cocyte):
Missile 1 has a final to hit of 92% and an AAM interception chance of 45%
Missile 2 has a final to hit of 67% and an AAM interception chance of 33%
I assume that no PD exists and that the 3v1 will be used for interceptions. I assume in both cases that only 1 interception chance is possible.
M1 = (1-0.45)^3*0.92 = 0.15 (15% chance to hit)
M2 = (1-0.33)^3*0.67 = 0.20 (20% chance to hit)
If the number of interception chances is different so M1 is intercepted twice and M2 only once then:
M1 = ((1-.45)^3)^2*0.92 = 0.025 (2.5% chance to hit)
If both are intercepted twice:
M2 = ((1-0.33)^3)^2*0.67 = 0.06 (6% chance to hit)
If you factor in PD then the chance of any missile getting through drops. If you then sum it up (to account for a typical salvo) though it becomes different, and substantially so if you say you can saturate their point defence or not, since in any case there isn't much difference in performance point defence wise so missiles intercepted by point defence fire will likely be eliminated.
Assume 8 missiles fired: 4 intercepted by AAM fire; 4 not:
Number of M1 hitting target = 4*.92 + 4*0.15 = 4.28
Number of M2 hitting target = 4*.67 + 4*0.20 = 3.48
At the end of the day the performance between the two missiles isn't exactly overwhelming different (for that particular situation anyway). Those two numbers aren't going to look too much different in a battle since they are the mean and there is the deviation to account for.
But what I hope I have shown is that it is not a straighforward thing. When you end up multiplying different probabilities together what may look like big differences tend to vanish. They may come back in the standard deviation...plus I'm not sure at all what that to hit distribution function would look like (it is not likely to be guassian for example).
Also by choice of numbers in an example I can skew the results I present. This is an unfortunate problem with this kind of analysis...it is a bias that anyone reading this stuff has to keep in mind.