Absolutely, PD ships should target largest missiles first.
There is absolutely no possible logical reason why a beam PD weapon would attack smaller missiles first. And at the tech level involved, it would also make no sense to introduce randomness. The largest missile is surely easily recognizable. If it is not, then there is electronic warfare involved. ECM, ECCM or anything else that could be added later on.
But barring that, largest missiles have to be targeted first. In fact, I really want this because I have been in a few situations and the AMM reached first, thus some ASM hit my ships.
Now, about the mechanical in-game means for that to happen, I'd say that what you are thinking could work.
Missiles with a missile target move first
Then all other missiles, from largest to smallest.
Seems pretty easy to understand to me, and functional as well.