Probably the most neglected aspect of playing guitar or bass is the tone. What is tone? Tone is your sound!
Sound is a chain, and it all starts at you. If you have a great guitar, nice cables, good pedals and a decent amp and you still sound bad, chances are it's you...sorry. Don't worry, though, this is also the least expensive problem to fix.
So practice. Run scales up and down, sit in front of the TV while fretting uncomfortable chords, work on your favorite cover song - all of this will improve your tone.
Once you're smoking, though, you still have to worry about your guitar. There are obviously hundreds of guitars to choose from, but the best guitar is the one that is best for you. Gibsons are good for jazz and rock, Fenders are classic blues guitars. Now you have two choices - Gibson or Fender - but remember, there are other brands out there. If you like the Stratocaster, for instance, consider playing some of the competing brands' versions of the axe - there are thousands of imitators. The fact is, though, some of these immitators are pretty darn good. Don't be too hasty! Even though you probably hate doing so as much as I do - shop! It will pay off. I own a Yamaha, a Washburn, a Takamine, and Ovation and two Fenders.
But once you have picked out a nice guitar, you still need a good amp. I suggest tubes. When it comes to tube amps, the classic answer is the halfstack/fullstack, where you buy a head (the amp itself) and cabinets (the speakers) seperately. There is Marshall, Peavey, Crate, Ampeg, Laney, Mesa/Boogie, etc. Everyone knows about Marshall, so they tend to be pretty popular, but don't neglect those other brands. I play out of a Peavey 5150 myself, and I love what it does for my leads. The clean tone is also powerful. Granted, the heavy rock tone leaves a lot to be desired, but it's not a heavy rock amp - it's a lead amp (which is why Van Halen uses one.) Hence, leads sound awesome. If you are more into Cannibal Corpse, you probably want a Marshall, although some Crate heads have a pretty brutal distortion channel. But in truth, all of these amps are good, you just have to pick the one you like - just like you did for your guitar. Laney, Marshall, Ampeg - they all work the same, but give you different flavors, if you like, of your tone. Like I said, Peavey's aren't much good for death metal, but they rock for other stuff.
The other thing to consider is the cables. You can buy dimestore cables for $.03/meter, or you can buy Monster Cables at $36/centimeter, but in truth wire quality comes down to two factors - wire gauge and the connectors. You can buy Monster Cable and solder it up with cheap solder, and you've successfully wasted money. Here is the logic of the signal chain:
Good-good-good-CRAP-good-good-good = CRAP!
If you have anything bad in the line anywhere, there goes your tone. One bad pedal? Crappy tone. One cheap solder point? Crappy tone.
That's the frustrating part. If you are technically savvy like I am, you'll find yourself making your own cables. While Monster Cable is a higher grade of metal, the fact is that the wire gauge can gain you as much or more as metal quality. I use 10 gauge wire, which is roughly the size of a lamp cord. That's a little big, but then so is my tone.
So my advice is, buy good cables (which is pretty expensive) or build your own. If you build your own, don't neglect anything!! Remember the signal chain. Buy good 1/4" connectors, good silver bearing solder - which is really not that expensive - and nice thick wire and you'll be fine. Some guitarists disagree with me and try to say that you need specially treated wire and gold plated connectors, etc. Fine if you can afford that, good for you. I can't. If you can't afford wire, see if you can scrounge up a little scrap cable from broken lamps, your dad's workshop - wherever. Just don't break the law! ;)
The final step to a good tone is pedals. This is really an "optional" category, as your best tone, technically, is that with the shortest signal chain:
guitar-cable-amp > guitar-cable-wah wah-cable-amp
...but who can deny that pedals are too damn cool to pass up?! I know I can't resist. So I stick them in my signal chain.
Fortunately, the negative effects on your tone will be minimal if you only use one or two pedals - and if you use good pedals this difference will be miniscule compared with what those amazing pedals do.
So invest wisely. Electro-Harmonix is my favorite manufacturer - they build miniature tanks that work every time and don't even drain your batteries when you're not playing! That's cool.
Other good pedals are Boss, Digitech and CryBaby. Of course this list can never be complete, but it is a good start. These are higher end pedals, so they will cost you more, but it is worth it! Trust me, you will thank yourself for spending the extra $20-$30 on good pedals. A) they will never break on you and B) they will do wonders for your sound.
I don't suggest multi-effects processors, though. Too much for too little. Too much can be a very bad thing. Reverb/Delay are basically the same thing, so combination pedals are ok. Generally, though (unless you spend like $350) multi-effects are just a lot of cheap effects. You're better off with a few good ones.
Often, too, effects can mask physical problems in your playing, and can actually stunt your musical growth if you over-use them. Using delay all the time, for instance, will not help you develop timing! Get to know your guitar naked. Unless you feel comfortable playing with your naked tone, don't dress it up. Practice instead.
Beyond that you don't have much to worry about. One thing to consider, though, is acoustics. If your amp is too boomy, try taking it out of the corners of the room. Too thin? Try moving the amp INTO one of the corners. Common sense applies as well - point the speakers toward you, but think about your bandmates as well. If you have a fullstack, think about splitting it in half. One cabinet for you, the other for your drummer and bass player. This will increase your percieved volume so you can actually drive LESS acoustic energy into the room. Whatever acoustic problems you're up against, they will be less significant when you are quieter.
But even all of this won't get you all the way to where you need to be. In my studio, the guitar sounds fantastic (I think) on tape, but kind of lackluster in the room - and I don't care. It's supposed to sound good on tape. On the other hand, when I play live, I don't care what it sounds like through headphones or on tape - I care what it sounds like out there in the audience.
It's all about who you are playing for. Many players have wonderful tones in thier bedrooms, then they go jam with a drummer and it sounds like crap. Happens all the time - why?! Because that's a bedroom tone, not a jamspace tone. You have to tailor your tone to the room you're in, the style you play and the volume level you play at.
Beyond all of this, though, experience is always the most beneficial addition to your rig. That means you've got to play!