I also have an 84 Camry, and it does this. It does it less (or I notice it less) than it did when I got it a couple of years ago, and this is partly due (I suspect) to having thorougly cleaned the throttle body with a spray can. I've also checked the throttle position sensor and adjusted the idle speed, and I suspect the culprit for any remaining hesitation is probably wear in the throttle butterfly valve. Or something else I really can't be bothered dealing with yet.

I believe there are other, more informative, threads regarding similar issues, if you search a little.