I'm starting to notice a lot of dirt and grime allocating on the underside of my strings. Is there anything I can just clean the dirt with? Like the green scrubby side of a sponge? Not get it wet or anything since that'll probably be bad for the wood.

I just got laid off so I'm trying to scrimp and save where I can. Unfortunately that also goes for my guitaring stuff.
Lemon oil works like a charm but be sure to buy the right stuff.
Peavey 3120
Mesa 2x12 Rectifier Cabinet
Jackson RR5
Jackson DKMG
LTD EC-1000
TC Electronic Polytune
MXR 10-band EQ
EHX Small Clone
Dunlop 535Q Crybaby
Voodoo Lab Pedal Power 2+
For Rosewood fingerboards we suggest removing the
strings and applying lemon oil to a cloth, then working the
oil into your fingerboard. Let the oil soak in for a few min-
utes before wiping off the excess. This should be done at
least once a year. Doing so not only cleans and avoids
cracking it also keeps your fingerboard from losing its lus-
Generally, maple fingerboards are pretty much mainte-
nance free. However if the neck starts feeling sticky you
can use a slightly damp polishing cloth without any guitar
cleaner on it followed by a dry cloth.

Source: Godin Performance series manual. May want to check the manual for your guitar.
If you do use lemon oil (only do so for rosewood fretboards - there are a few other more exotic woods that can benefit from it too, but the most common is rosewood. Do not use on on fretboards that have a finish on them, like maple), remember to only use the smallest drop each time, and keep it away from the back of the guitar's neck which probbaly has a finish on it.

Also, don't use lemon oil if the board is already naturally oily and in decent condition. Sure, it may be dirty, but over-oiling a fretboard that doesn't need it can lead to permanent warping. If a rosewood board is looking grey and actually worn-out, it may need to be treated with lemon oil; however a lot of better quality rosewood can last for decades without ever needing to be treated. Additionally, the quality of the natural oils in your skin and how much you use the guitar can also greatly effect the condition of the board and if it needs treatment or not.

If the fretboard is actually in fine condition and you just want to clean off the visible dirt and dust, just get a dry, inkless microfibre cloth (very cheap and worth buying a pack of them, they're invaluable for cleaning just about anything) and just give it a dust-off with that.
Yes, I know everything. No, I can't play worth a damn.
A child is trafficked and sold for sex slavery every 30 seconds. Support Love146.