Another thing could be a limit on the number of missiles that can be handled by a single fire control.
I like this one. It's quite realistic (up until fairly recently, this was a major limitation on naval SAMs) and it would cut the problem immediately. It also fits the general principle of solving problems with the least new stuff possible.
Even better would be having the accuracy of missiles go down as more are stacked onto the same fire control. Time-sharing the FC (different salvos linked to the same FC having different targets) might cut it even more, although this would muck up defensive AMM use considerably.
Even with ablative armor (say damage of strength 1 is negated) there would still be the issue that anti-missile measures can be largely negated by simply putting all missiles into a "super" volley.
How? If I take 100 strength-1 AMMs but have ignore-1 armor, all of them will bounce off.
Another solution is to add a mechanism for proximity kills. Say that a sufficiently strong (4-9?) AMM warhead can do damage to other weapons in the same salvo as the target. Suddenly a (probably ASM-sized) AMM can counter an arbitrarily-large salvo, and there's a serious motivation to go with lots of little salvos. Again, no major new mechanics.
Edit:
Thinking proximity kills over more, I see a problem. Allowing a missile that auto-kills a whole salvo is going to dramatically change the way the AMM game is played, and I'm not sure that's a good thing. There are two ways to deal with this:
1. The warhead is not guaranteed to inflict proximity kills. Instead, after the missile itself hits, there is a chance of a proximity kill on each missile in the rest of the salvo, based on warhead yield. I'd have to run numbers before making a recommendation on the values to put there.
2. Increase the chances of a proxy kill based on how many missiles are in the salvo. Assume that for (mumble mumble transnewtonian) reasons each salvo can only occupy a definite volume of space, which is larger than the damage radius of a typical weapon, but not hugely larger. With a small salvo, the missiles are spread out, making it unlikely that more than one or two would be caught in a given weapon. As more missiles are added, the chance of a given missile being caught in the radius goes up, although it's going to plateau at some point as the percentage of the space that the damage volume makes up. (This can be somewhat modified by making the AMM smart and having it go after the densest part of the salvo.)
Edit 2:
Actually, I don't think 2 is necessary. If the proxy kills are percentage-based, that provides all the incentive needed to avoid large salvos. Let's assume that proxy kills first have to attack a missile using the normal rules before triggering the proxy effect. (An alternative is to have them only attack by proxy, and give them a bonus to 'attack' necessary to trigger it.) We have two options, both with the same probability of hitting:
A. A size 1 AMM.
B. A size 4 AMM with a 25% chance of proximity kill if it hits.
Let's say we choose between a single B and 4 As, options which should have equal cost. (I'm assuming that all weapons hit. Alternatively, assume I fired 1/PH of B and 4/PH of A.)
If the target salvo is 5 weapons, A will kill 4, and B will kill one directly and one by proxy. If it's 9, you'll see 4 kills by A and 3 by B. Crossover is at 13, and above that, B gets more kills than A. If we assume that above size 4, the scaling of proximity kills is linear with warhead size, you see the same effect. Nonlinear scaling there is going to move equilibrium about some, but it's not hard to make a scenario where it's better to shoot current AMMs at small salvos and proxy-fused AMMs at big salvos.
This does bring up the launcher paradox. At the moment, the launcher system has two separate linear scaling effects with weapons size, size of the launcher and time to reload, which means that the 'throw weight' (MSP/unit time) is inversely proportional to the size of the missiles being fired. A 6-HS size-6 launcher is going to fire every 30 seconds (6 MSP/(30 seconds*6 HS) = 1/30 MSP/(HS*sec)) while a 1-HS size-1 launcher with the same tech fires every 5 seconds for 1 MSP/(5 sec*1 HS) = 1/5 MSP/(HS*sec). I'm not sure that there shouldn't be some scaling, but the current setup is grossly against bigger launchers, and that probably plays into the reason AMM salvos are so powerful.