I have had to set the action very, very high to reduce fret buzz.

I've cranked out the adjustments at the bridge so that the action is perhaps 1/4" high at the 20th fret, but at the first fret the string is almost touching (low E string).

The Low E string buzzes against the 2nd fret Im pretty sure.

Is this a worn nut or does the truss rod need adjustment.

I'm doubting it is a worn nut. What's your guitar, if it's a cheap, supermarket guitar, maybe get a new one, there's hardly any real point getting it fixed.
Cam Sampbell's my hero