Specific details would really help. Things like a full ship design listing, range setting for the area defense, active sensors if not from the defending ship, size and speed of missiles, etc etc etc.
Are you still playing v5.42? or should this be in the v5.5 bug thread?
Unfortunately I finished my random testing and deleted the database to upgrade to v5.54 (I was using 5.42). However I can specify most of the pertinent statistics from memory (and the second ship was identical to the first, so no sensors or anything changed). I SSMed up the full beam/sensor etc etc technology tree, and was shooting very slow missiles at them to get an enormous range advantage on the missiles.
The ship had the following data relevant components:
An enormous resolution 1 active sensor. All technology unlocked, 35HS of space, it could see size 6 missiles and smaller at over 400m km. None of the ships on either side had passive sensors, so if the "Big Target" could see a missile it was because it had an active sensor contact. The firing ship was close enough so that missiles of any size were spotted immediately.
A large fire control, built with a targeting range of 1.4m km, the max. I can't really remember the target speed, whatever the max research is times 4.
A quad turret of lasers that rotated at a speed close to the fire control speed. The mounted lasers were (I believe) the largest lasers that could fire every 5 seconds, they had a range of 340k km.
The ship had a lot of Cargo Spaces to get very large, for my tests in damage configuration for laser warheads, it also had lots of armor (50 thickness) to survive many test rounds before I needed to SM repair it.
I did several rounds of various sorts of tests, the first intentional hits (speed 1) with a few types of warheads and laser warheads to determine damage spread, then I upped the speed to determine if the different laser beams in a large laser warhead targeted separately.
At this point I moved to targeting tests. I redesigned a few types of missiles for these, the size 3 "Slow Boom" which only went 8.4k km/s, matched by the size 3 "Slow Laser". Later on, to see whether the small sizes had anything to do with it I also introduced a size 40 drone "Slow Boom" and a size 40 drone "Slow Laser" that each moved 12.6k km/s. In every test I just fired one missile at a time.
I started off with various manual targeting, trying both Auto Fire, and not, manually firing, all sorts of things. It's at this point that I think I must have put this ship in an invisible bad state that caused me problems, I don't know if I told it to target a ship out of range, or target a missile that then hit it and exploded, or what (and this is all hindsight, so I didn't try to do anything to clear the targets) but by the time I put it into Area Defense Mode (range 30, or 300k km, short of all of the relevant maximum ranges but with more than 300 seconds to shoot at the slower missiles) it just didn't attempt to shoot at anything regardless of range. It also didn't do a final PD mode when the missile did hit it. Final Point Defense mode worked totally fine, but Area mode didn't work at all. As a final try I sent the size 40 "Slow Boom" after it, with a strength 900 warhead. The resulting explosion completely destroyed the ship.
I SMed in a fresh ship as a Hail Mary, and from that point on the Area Defense Mode worked completely normally.