Hey Nick,
I'm driving the same car (Japan built XLE 03) and have the same problem. The sound comes from the A/C magnetic clutch, a very weird metallic sound can be heard when it's turned on. It gets worse on freeway, the faster you drive the louder the sound is inside the car, besides you can feel a kind of a weird loss of power each time the clutch is turned on. It is not a constant sound, it appears when you turn on the A/C or when it's on automatic and turns itself on each time to maintain the preset temperature. The V-belt is not the reason for, it has been replaced and is brand new.
The higher the RPM the louder it gets. My dealer says there is nothing wrong and the compressor sounds fine, but there was never such a loud metallic noise in the past (as well as not such a significant power loss, feels like a strong wind blow hitting you in front of your car). I have looked on the internet and found this TSB.
http://www.toyotapart.com/2002-2003_...T-AC001-03.pdf
The problem is I do not live in such area with "hot high humidity conditions". Have you solved your problem? I can't believe I have to replace the compressor with its built magnetic clutch, this is not very Toyota-like you know