I found a book called Sight-Read any Rythym Instantly, and its so helpful. However, in this book it said that counting syllables (1-e-and-a, ect.) in between quarter notes is counterproductive, and that one should memorize the sounds of x number of beats per note. The idea is that you can keep time better because you only have to focus on the quarter note beat.

At first I didn't like this idea one bit, as I am used to counting out every note. But my sight reading is also pretty ****ty. I was talking to my freind, who is a drummer and he says he never counts out his music past quarter notes or eighth notes. But, hes also very lazy. How do you count out music? Do you count out every note, or are you less detailed than that?
It really depends on the time signature, and the amount of syncopation involved. I've gotten used to the idea of "fast" notes and I don't have to count out every single one. In 6-8 time, I count 1-2-3-1-2-3. In 4-4 I count either 1-2-3-4 or 1 & 2 & 3 & 4 &.
When I'm reading something with a difficult rhythm for the first time, I usually count out every note slowly and speed it up and practice it until I get to the point where I don't have to count anything past quarter notes. I don't personally think it's counterproductive to break down the rhythm of every note in a piece at first. I think that doing this actually helps you memorize the sound of X notes per beat so you eventually don't have to break everything down.
...to give your love no matter what is what she said...