According to the '06 owner's manual, it's only supposed to be finger tight, comes right off. FWIW, I never take mine off when I go thru the car wash, never been a problem and mine is just finger tight. I'd be more worried about the spare tire cover if you don't have a limited, it can be blown off by the high velocity dryer, happened to mine in one wash that I don't use any more. It's been reported several times at Rav4World.
http://rav4world.com/forums/viewtopic.php?t=8345&postdays=0&postorder=asc&start=0
http://rav4world.com/forums/viewtopic.php?t=8345&postdays=0&postorder=asc&start=0