I have a peavey vypyr 15 watt and a fender squire and am using the headphone jack. I only use the 6505 amp model setting on the vypyr and as much as I try tweaking everything I keep getting a muddy distortion and just sounds terrible. The gain on this amp just sounds atrocious compared to Amplitube , a software amp plug in for pc.

what can I do? I also tried using a les paul with the vypyr and it sounds like crap as well.
Well to me the 6505 amp model sounds crap. Its very muddy and too heavy on the bass. I use the K-stein then add some bass and mids to the eq. And i have the 30 watt version. It could be your cable aswell, my old lead (was a freebe) was shit and cut my sustain and treble. Also twaek you mids, treble and guitar tone knob to reduce the muddyness.