I know this is a modeling amp, but how do effects pedals sound when played through this amp? Is it worth by a pedal? I dislike the tone. I feel this distortion on this amp is too muddy.
The most likely answer is you need a new amp. My question is, what pickups are you using? If they are muddy like the stock ones on my ibanez then a new amp will still sound muddy.
Washburn MG-44(E)
Ibanez RG421 (Eb)
Art & Lutherie Electric Cutaway
Vox Valvetronix VT40
Vox AC4tv 1x10
Vox Original Wah-Wah Pedal V847-A
MXR '78 Custom Badass Distortion
I wouldn't buy a pedal for it. It won't help imo.

Have you used the software? Changing the cab sim can help a lot. You have more tonal control as well. You have a mid knob for example with the software that's not controlable at the amp on it's own.
I believe they are alnico p90 pickups. Its an Epiphone 1961 SG special. Is the software actually good. and why not get a pedal?
The Mustang 1 does not work as an amp on its own. The whole reason it exists is to use presets created in Fuse and assigned to banks on the amp. The stock presets on the Mustang 1 are worthless garbage. Install Fuse and start downloading presets from the internet; there are some great tones. Or just create your own by starting with only an amp, knobs at 5, and make adjustments from there. This amp can produce some solid metal tones; just ask Ola: https://www.youtube.com/watch?v=jvtvstFjVuU