My band would like to play johnny b. goode for a song. The only problems are:

1) What should the keyboardist play
2)I am the guitarist and how should i set my amp - i usually have:
Gain:3-5, Bass:8 Contour:5 Treble:5Reverb: 3-5
Gain:4 Bass:5 Contour:3-6 Treble:10
(Sounding like Hendrix and Page, respectively)

What should i do about these things?
I'll pretend I can mod your amp but break it instead.
setting two with more contour and very little reverb

and if your keyboardist is good, i'm sure he can write his own fills for it
There's a lot of piano in Chuck Berry music. Just have him play the chords (think rhythm guitar) and periodically throw in some fills. Like the guy above said, he should be able to come up with something.
Depends on how you're covering the song really, if you are playing it in the original style, naturaly overdrive your amp without using any effects peddles and have the keyboardist play a boogie woogie style backing in the appropriate keys.