Personally I don't PVP, but in PVE, hands down the most annoying spell for me is spirit armor. When a mob suddenly has 400 more hits than normal it can really put a damper in a deck that's tuned down for farming.
If you are having trouble with the Spirit Armors, make friends with an ice wizard. They will help and can easily steal those armors for themselves. That's what I do.
Hmmm good question. In my opinion the most annoying is weakness. Especially when the enemy goes first. So boom you have a lot of tings to power up your attack it should be just enough maybe a little too much. Then they use weakness and you waste all your pips and dont kil anything. it just gets annoying. But yah spirit armor is pretty annoying too. I just dont find it as often as weakness.
I have to admit, Spirit Armor is a pain. On a day to day basis, I hate the spell that Balance Monsters use frequently and that is the one that reduces attack damage by 25%. I don't know how many times I will have already selected a spell that produces enough damage to finish a monster off, only to have that annoying spell cast on me so that I have to go another round.