What catches my attention here is the fact that cylinders 1, 3 and 5 are all on the same bank. rear bank as I figured out. (this one's a bitch to work on

)
I'd check/test if the rear O2 sensor is working correctly. It could be sending the wrong information to the ECU.
Also check the wires from the coils, are they properly connected to the coils, corrosion, resistance check, if you've got small enough hands to work on the rear spark plugs.
Could be a timing problem too, like the timing belt having jumped a few teeth on the rear bank camshaft gear... all cylinders on that bank would probably be misfiring while the other would just work fine. And even when not misfiring, rough running would be expected.
http://www.**********s.com/camry/eg2.pdf
O2 sensors : Page 306+
Timing : see timing belt installation from page 49.
Feel free to browse, detailed engine service manual for 1MZ-FE. There might be a few differences between yours and the one in the manual (earlier version, mid-90, previous gen Camry) but it should be 99% the exact same.
EDITED : cylinders 1, 3 and 5 are on the rear cylinder bank. I was mistaken in the first place.