Okay, my friend is a guitar teacher. He is well versed in theory, scales and modes. We jammed for a while. He played some modes that sounded like garbage! So I broke out my Guitar Grimoire and tried some of the modes they had listed. Again it sounded like garbage. There would be a few notes in each mode that sounded FAR out of place. As if they had gone bad sitting on the window sill all day. Why then do some guitarists use these ugly sounding modes?
generally they are meant to make these sort of strange sounds.. But they are made to fit into the song. The notes that sound "off" are always purposely done to fit in. Do you know exactly which scales they were???
The one he did that sounded awful to me and really stuck out was the first position of the E Lydian Augmented. He also used the B flat Phrygian, which sounded odd in a bad way. Maybe my ears just weren't used to hearing them. But the off notes didn't sound as good, as say an egyptian off note style.