I play mainly shred like Vai and Gilbert, and metal like Dream Theater. I have an RG321mh, and I want to know if it's worth upgrading or not? I don't want to spend a whole pile of money on it if it would be better to buy a whole new guitar.

I'm not so sure about the sound of it atm, but im pretty sure its the ****ty stock pickups in it.

I'm considering installing some dimarzios in it, and a pickguard for looks.
I would change the pickups, yeah. The body is mahogany, so I'd say it's worth it. If it was agathis or basswood, I'd have said just get a better one. While you're at it, find a good tech and get the frets crowned, leveled, and get the action & intonation set. Getting the frets crowned and leveled makes a world of difference in the feel and how it plays.
^^ listen to him.

It's a mahogany body which means it would be really nice to fix up.
Put different pots, volume tone knobs, different paint job, locking tuners, new pickups, whatever you want.

The dimarzios sound like what you want.
as long as your happy with the body and neck then you could make the guitar great
diffenitly upgrade the pickups first as that will make the biggest and best difference
Epiphone Les Paul Special 2
ESP EX-400
Roland Cube-15
Bugera 6260 head
Home made 2x12 loaded with celestion G12t-100s