Anyone ever heard of replacing a pickup selector switch
with a dimmer switch?

In theory it would open up a rainbow of tones...

but would it actually work?
I bet Charlie Brown's teacher's name was Mrs.Hammett
I haven't heard of it, and I personally wouldn't know if that could work.

However, if you want to open up a new range of tones, try a rotary pickup selector, like PRS uses. You can wire up your pickups to be in series/parallel/out of phase/whatever else.
Ibanez RGA121 | ESP LTD H-1000
Axe-FX Standard