When was the last time you changed the fluid and filter?
My Es250 was having some trouble shifting when cold, and when it was warm it shifted harsh and would slip in and out of gears.
I changed the fluid and filter and it all cleared up, super smooth, crisp shifts, but after years of driving only manual cars I forgot about the tranny cooler in the radiator

and now I need to do it all over again.
I would suggest inspecting your fluid. If it not red, change it and the filter, clean the pan out, MAKE SURE YOU GET A NEW GASKET!!!! See if that clears it up, drive it for a few hundred miles and change just the fluid again.
ALSO, give this manual a read. Make sure you go through every page of it. Im sure there will be something in it to help you diagnose this problem. Good luck to you.
http://camrystuff.com/manuals/Gen2/A..._Transaxle.pdf