Unfortunately this isn't limited to just 48 hour spells in my experience. I have 2 gardens and both experience *huge* amounts of lag when I do anything to any of them as far as taking care of their needs.
To avoid gardening lag I go to the garden, then leave, then go back. Casting a spell then works normally.
If I forget to leave then I make sure the plants are not in my immediate view (turn to a wall etc) and then it lags less and resolves faster. The above method is the best method I've found though. Adding teleports can make this faster if you'd otherwise have to wait on the 30s teleport cooldown.
