I have an '08 Yaris LB and my brother has a '91 Subaru Legacy AWD Turbo sedan.
His car handles 10 times better than my Yaris, and we both just have springs and aftermarket wheels as far as suspension/handling mods.
Not saying the Yaris isn't a good handling car, as I am overall impressed. Just saying I know in our case, his Subaru eats my Yaris's handling up.
The Yaris is a decently quick car for how small the engine is. All I have done engine-wise is a TRD intake, Nitto axle-back exhaust, and a TRD quickshifter/short shifter kit.
And I'd like to point out, mod-wise. There is a decent amount of aftermarket support, but most of it is pretty expensive in my opinion.
Overall, I love my Yaris and plan to own it for at least 5-10 more years.