2003 HL V6 AWD Base 332k miles.
I have to give myself a C- regarding my diagnostic skills on this one. The car had been gradually getting worse with a noise at the rear. Been going on (intermittently) for some six months. At first I was convinced it was a wheel bearing and even bought the parts to fix it. Nope. Not a wheel bearing.
Then investigated the brake caliper. It was "rattle-y" but no, that was not it.
I even replaced the sway bar rubber bushings, that seemed to help but no, that was not the cause.
Then figured it might be a loose exhaust component. No, everything was tight back there. Hum...
Then I remembered seeing a random YouTube video about a 2003 HL with a phantom noise just like I had.
Yep. Left rearward (aft) lateral control arm bushing! Replaced the arm and cured the rattle. Yay!
BTW-if you are facing the same thing be aware there are a couple of different configurations and styles of control arms depending upon year of your vehicle. It is a little confusing when looking for parts. The 2001-2003 (and 1999-2003 Lexus RX300) uses the simpler "ring-end" type arm. Both ends of the arm have ring configurations. The bushings at each end are part of the arm itself and are not replaceable. (Later versions use an arm with a ring end at the frame end and an open "fork" at the knuckle (carrier) end. The bushing resides in the knuckle. The bushing is replaceable.)
Anyway, the point is my 2003 AWD required a whole new arm. Dorman, Febest and Toyota sell the arms. I used the Dorman.
2nd BTW- Plan on having to cut out the bolt at the frame end. (Side grinder.) This end contains the adjuster cam bolt that allows setting of the wheel toe-in/out. Mine was rusted tight and not reusable.
I replaced the cam bolt with a grade 8.8 regular M12 bolt and nylock nut (and flat washers). I also had to fabricate a small metal bushing to take the place of the inner adjuster mechanism. Good luck finding a replacement adjuster bolt kit. Haya....