I recently bought an adapter for my car so I can plug stuff in using the cigarette lighter as seen here . I have a 20 watt guitar combo amp that I have played using this. Do you think this is harmful to the car, the adapter, or my amp?

if it works you should be fine, that is what an inverter is for.
2002 PRS CE22
2013 G&L ASAT Deluxe
2009 Epiphone G-400 (SH-4)
Marshall JCM2000 DSL100
Krank 1980 Jr 20watt
Krank Rev 4x12 (eminence V12)
GFS Greenie/Digitech Bad Monkey
Morley Bad Horsie 2
MXR Smart Gate
The only way to be certain is to check current ratings. the adapter should have a current rating. the car's power source should have a current rating and the the amp should have a current rating.

If the amp draws less current than the other two are rated for then you're good to go.

My wife plugged a crockpot into her car once. Then there was smoke and a smell of burning plastic. That's what might happen if you try to draw too much current.
Back in high school I played jingle bells in the back a pickup truck in a christmas parade with my old Spider III halfstack which was like 150 watts. It was about 30 minutes long and nothing bad happened to either the truck or the amp. It should *most likely* be fine. Do check your current ratings though.
My son often uses an inverter when he goes to the lake, and I've used one a few times when I was camping.

Guitar amps don't really take that much power, even tube amps. I've got a little 300w inverter that's done the job admirably.
“Ignorance more frequently begets confidence than does knowledge.”
Charles Darwin
i have used amps with my 120V car adapter, mine is rated for about 400 watts though. the one you list is only rated for about 80 watts.

but to give you an idea, a bugera V22 consumes about 72 watts (i will assume that is with the amp pretty cranked out) and that is about a 20 watt amp. i would imagine a solid state amp probably uses less power to run it. either way it should probably work.
punk isn't dead, it's always smelled that way.

"A perfection of means, and confusion of aims, seems to be our main problem."