I'm not sure if this question has been answered before, but when I play I tend to get a quiet (but noticeable and quite distracting) "hmmmmmm" ing from the string's I'm not hitting at the moment while playing. I like to tape myself playing and play it back to help improve, and I notice it A LOT there.

It goes away if I mute the strings but that makes playing notes A LOT harder. When I turn my amp up on dirty/OD channels, it turns into a huge SSSSSSQQQQQQQQQUUUUUUUUUEEEEEEEEEAAAAAAAALLLLLLL as soon as the volume goes up, and I usually can't stop it by muting the strings like I can at lower volumes. These problems tend to only happen while on overdrive, or clean with high gain though.

My guitar is an Epiphone SG with the original bridge, Ernie Ball strings (.10s), a Radioshack cable and a Peavey Vypyr amp. I'm also not sure if this helps at all, but I only use the bridge Humbucker most of the time unless I'm playing chords.

I was thinking of maybe putting little bits of duct tape under the grooves in the nut to help dampen extra vibrations, but I'm not sure if this would interfere with playing open notes or not.
Put a hair band under the strings, behind the nut. It'll help a bit. Also, palm mute maybe?
Take a look down near the bottom of the guitar where the strings are anchored down (as opposed to the tuning pegs up on the headstock). I don't know if this is your guitar exactly but it should be close enough for demonstration purposes:


Do you see the metal piece where the strings are anchored? There is another metal piece in between the anchor panel and the first pickup. Check to make sure the strings are properly aligned on THAT piece because, if they're not, an annoying buzz/hum can occur when the strings are left untouched with the amplifier turned on.

Did you string the guitar yourself? If so, are you proficient at it? It's easy to make a mistake (even for seasoned guitarists) so just double-check the work to make sure everything's on the up.
How long have you been playing? You just have to develop a technique for dampening the open strings. When playing power chords, I use the flat part of my hand or fingers to damp the higher strings. While soloing, the palm of my right hand dampens the lower strings. If soloing on lower strings, I damp the higher strings similar to playing power chords. Tricky, yes, but you're on the right track by listening to yourself and recognizing what needs attention. Someday you will do it by second nature without thinking about it. If you play in a small room distortion and feedback can do funny things too.
It goes away if I mute the strings but that makes playing notes A LOT harder.

Tough noogies. Slow down, work on your technique and don't stand so close to the amp.