I think the only thing is you need to draw often enough that whatever lessons you have learned the last time you held a pencil don't evaporate before the next time you hold a pencil. Repetition is important to learning any skill that requires motor skills or dexterity, and the time between those repetitions directly effects how well you learn.
It's like anything else. If you don't do it for a while, your skills get a little rusty. If you know something really well and don't do it for a long time, the first time you try it again, you won't be totally up to snuff. There's some refresh time required. If there's a long gap between your drawing sessions, then you'll spend just as much time catching up to where you were as you will improving.
Three hours a day might be a bit much. But try to practice enough that you don't get rusty so all your drawing time is spent improving rather than catching up to yourself. How much practice that actually is varies a lot depending on the person. Only you can tell if you're gaining something or just getting yourself frustrated.
In the immortal words of Socrates, who said, "I drank what?"