Keep in mind, its a 4cyl. SuperKaioken, a dude with a black SE V6, has a Borla exhaust (same as TRD) and his sounds pretty similar, just not quite as loud.
In terms of ride quality after lowering... well, of course its going to be worse. I can't deny that or else I'd be bullshitting straight through my teeth. But as far as how much worse? Well, thats partly up to the individual. For me, I like it-- tighter turning, less body roll, and nicer look. I like the way it feels when I drive. But don't get me wrong... I have a stock '07 V6 LE and a stock '06 RX330, and there are days where I love to drive 'em because the suspension is just so smooth.
Its really up to you. If its something you might be into, then yeah, lowering is for sure the way to go. Some people don't like it. *shrug* To each their own.