Lactic acid dosn't build up chronically. As far as I know, it's metabolized fairly quickly. As soon as the burning is gone; the acid is gone.
I've been told that the soreness in your muscles after a workout is due to the fact that the blood vessels in your muscles expand greatly after an anaerobic workout to get oxygen and nutrients to these areas quickly (this is also knwon as "getting swoll"). After a quick initial expansion, it takes a long time to contract again, and during that time, blood creeps into your tissue and... other... bad things... happen too. Anyways, the punchline is that if you take a cold shower after a workout, the bloodvessels wil contract much faster and prevent soreness. It works like a charm for me, and I've never had problems with getting a good recovery.