I'm doing a blues solo and the chords are A D E. Im using Am pentatonic and A Dorian. But I've heard that you can use D minor and E minor when the backing track switches chords. But when I do it it sounds good on the D chord but when it goes back to the A it feels as if Am pent doesn't sound right there... Any ideas why?

Are they simply A D and E? Or are they dominant?

If they're dominant, you can play Mixolydian over each chord, the A blues scale (or a different blues scale over each chord), or a combination of those scales (not simultaneously, obviously). Dorian is also an option as long as you don't hang onto the natural 6th too long.

Part of the 7-string Legion

Check out my profile
and my 7-string Ernie Ball MM JP-7 build

What you really want to use over the D and E is major. Specifically:
D and E major pentatonic, D and E major arpeggios, or D and E mixolydian.

Even over the A, the same sorts of major is very strong. Because it's blues, an A minor
scale works over the A (the A Blues scale is essentially minor). That's what gives it
a blues sound. But, you're not really using it as a "minor scale", so complementing
it with some A major will tend to give it a more melodic sound.