I have had Lexus LS-series and driven Camrys, new Buicks (Lucerne etc.), BMW7-series, Audi A6, Jaguar XJR etc. I'd say Avalon is very quiet, but can't beat Lexus LS-series. They are eeriely quiet. However, it is well comparable to ES-series and seems to have more interior space.
To me, closest to Avalon (XLS) is oddly(?) the new Buick Lucerne. I sometimes get flashbacks of Camry too, but ship like size of Avalon and better materials makes the ride way more comfy. What I especially like in Avalon is that you barely feel any bumps or cracks on the road.. and we got plenty of those here in Bay area. Tires make a big difference, so if you put in those 19" super low profile Yokohamas, it's likely to make more noise.
Things that I don't like in Avalon are somewhat vague steering feel, pronounced body roll and torque steer. Steering feel probably derives from fat bushings and power steering that seems to be on steroids. Body roll and rocking are the price you have to pay for comfort. I have not driven any FWD with over 200 hp that doesn't have a significant torque steer. So, it's not a sports car - if you are looking for one, save your money and get Evo, WRX STi or M3 and learn how to drive.
Overall I really like Avalon. If you want to have an "undercover" luxury ride, roomy cabin, sufficient power and 25MPG++ it's definitely the way to go. JBL Synthesis does it's job very well, so get one with it if possible.