I have an MAB3 Dean guitar. It is a Floyd Rose. When I got it, it had 3 springs on the back and I put 5 because the I was using really thick strings (Dunlop guages 12-16-20-32-42-54). Now I want to try the Zakk Wylde custom which has guages .010, .013, .017, .036, .052, .060 is this too much tension for my floyd rose? I already maxed out the springs. Should I just go for it? Thanks!
It should be alright, but be away that you're raising the gauge of the lower strings but lowering it of the higher strings.. this is effectively "swapping" the tension around and could potentially warp the neck of your guitar!

As for the Floyd trem, I think it will handle those guages fine might have to remove a spring. See how it goes! If you can get the trem level/set up correctly and have the guitar in tune, then you're fine. I use four springs on my 7-string, so you should be more than ok