There are sound deadening materials, and there are sound absorption materials. High-end luxury cars have both.
The aluminum backed butyl sheets (I used killmat - same thing as more expensive dynamat at a fraction of the cost) are sound deadening. They will reduce the vibration in the metal, that tin-can rattling noise. You don't have to cover ALL the panels with it, 20-60% of metal is sufficient to achieve all the effect you can with this material. It will definitely have some effect, but it won't solve the problem completely. I took door cars off and put it on the door skin, trunk lid, rear quarters, spare wheel well, wheel tubs, rear deck.
Then you have sound absorbing materials. These are usually thicker, fabric type material that go on top of the butyl sheets. More effective, but also much more difficult to install because they're thicker, so more difficult to fit.
Then you have multi-layer acoustic glass and other stuff that's outside of reach of mere mortals.
Take a look at an ES300 or Avalon in a junkyard. Both were upper scale cars, so you may see where Toyota placed additional insulation in those cars to give you some ideas, or maybe some parts will even fit the Camry.
Ultimately though, the TRD exhaust may have to go. Frequent consequence of having a family and having to have more people in the car that don't enjoy the exhaust as much as you do.