I can sweep cleanly going down on my electric guitar. Coming back up usually gets a little messy. But I was just messing around on my acoustic and I could sweep up perfectly but sweeping down was a problem. I can sweep in one direction just fine, I only have the problem when I try to go up and down continuously. Is there anything in particular I should work on besides just doing it over and over again?
Just take it real slow and analyse your technique. Try and see what it is about your technique that allows you to sweep comfortably in one direction, and look out for what changes. Check that you're not digging in too much on the downstroke, and that your pick is striking the string at an angle (30-45º as hitting the string with the flat side of your, as either of these will slow your pick down and ruin the fluid motion.
Go back to practising a simple 2-string sweep, and once you can nail those consistently at tempo, then move onto 3-string sweeps, making sure you practise as many different shapes as possible, and continue till you're comfortable sweeping full 6-string shapes. Practise with both clean and distorted sounds to make sure that your coordination and fretting are perfect, as well as your right and left hand muting technique. It's pointless sweeping with poor coordination, or with open strings ringing left right and centre.
This process may well take weeks or months, but it's the only way to be able to do it smoothly and comfortably.

Martin Goulding is an example of perfect sweeping technique (and any other technique you can think of too):
That is the level of control and coordination you should aspire to.
He didn't pick when coming back up, he said he just did hammer-ons. But anyway. My right hand sweeps in the right tempo and very fluidly. The left hand is what trips me up. The action on the acoustic is 10 times higher than my electric guitars and the neck is also a lot thicker. I'm pretty sure those are part of the difficulty I'm having. I also didn't t take the normal route that most people take when tackling sweeping. I started with 5 string sweeps (less right hand work and much easier to find a flow). I can do 3 string sweeps cleanly now. The culprit is usually the D-string.
He didn't in that example, no, but that's probably harder. Also, if your pick is travelling upwards, that's a descending sweep.
If you're sure that your left hand is the problem, then mute the strings by holding the neck with your right hand behind your left, and practise hammering on every note up and down the shapes. Do that to a metronome, making sure the notes are absolutely even and that you can control it at all tempos. You should find that after practising that from 45 minutes or so, that things are far smoother when you bring your picking hand back into the equation. Do it regularly and you'll improve no end.