Besides bolting up, you'd have to think about engine mounts and electronics, because both of those were changed with Generation 5.

The throttle is now drive by wire, and the main mounts are now on the sides as opposed to back and front.

I suppose if you really wanted an easier manual swap you could swap to a 4 cylinder...

 

Not possible without modification. There is no DIY guide or 'How tos' for this. Manual transmissions ONLY came in the 2.4L Gen5 2AZFE for the Gen5 (2002-2006) Camry. All V6s were automatic. Mounting points for the Gen4 to Gen5 is significantly different, so I'm sure the transmission would be as well. Everything connecting the engine/trans to the car would have to nearly be custom if I had to guess.

 

Maxima and Jaguar X-type are V6 FWD with manuals.
Cadillac CTS and Lincoln LS are RWD.

 

The engine crankshaft might not have a provision for the pilot bushing for the transmission input shaft, so you might be dead in the water right off the bat. Then you'd have to figure out how to get the computer to not barf when it didn't get proper signaling from the automatic transmission. One way around that might be to go with the previous generation computer, but then the instrument cluster might not work (assuming that any of the harness plugs match--chances are you'd have to make your own frankenharness), and forget passing emissions if you're in a state that checks them. For mounting, the transaxle to cradle mount from a 2AZ may work, but that depends on whether Toyota designed the 1MZ mounts to keep the bellhousing flange in the same location as the 2AZ, relative to the car, so they wouldn't need to worry about stocking different-length halfshafts. And, of course, you'd need a bunch of other parts to make it work.

You would have been better served to get a 5-speed car and try to drop in the V6. Fewer parts to change.