somehow balance out my bands volume, and then somehow turn up all of the instruments at the same ratio so that you can hear them all, but at a different volume?
Mic everything, run it into a mixer...get the channels individually balanced, then raise/lower the master out. Thats what i think you mean anyway
You mean when practicing/gigging? Or when recording?

If you're recording you should have every instrument on a different track so you can control volume.

Otherwise just keeps messing with volume until you get the layering you want.