OK, I'm going to give you Camry guys a try as you seem to a little more technical than the RAV4 crowd.
I've got a 98 RAV4 - 3SFE - same as the Camry. Mine is manual, 107k.
It's got what sounds like a can of bolts rattling - only when warm and only at neutral throttle/load positions... That is, neither coasting or accelerating.
I thought it might be part of the timing belt assembly or water pump - perhaps a bearing failing, but I just had the entire timing belt, water pump, tensioner, spring, etc done and she's still rattling.
I can get it to do it at idle, bringing engine RPM up, and lifting off the throttle just a bit.
A clip of it:
http://lakebox.dnsalias.com/photos/1...4/MVI_1167.AVI
My next suspicion is cam shims, but I'd think those would rattle all the time - like any misadjusted OHV motor... And perhaps worse when cold.
Any guesses?