The current GS has yet to migrate to the tnga-L platform and rides on the older "New N" platform from 2012. Despite its age, that platform was exclusive to Lexus and more upmarket than the plebeian tnga-k platform that the current Avalon sits on, which accommodates everything from a $24k Camry all the way up to a Highlander and eventual all-new RX. The GS platform can also handle a 5.0L V8, again indicating its more higher market design. That being said, newer platforms usually improve rigidity and other handling characteristics, I would guess the newer tnga-k has closed the gap to the older "New N" platform, but I'm just guessing.
The biggest difference as you pointed out, will be the rear-wheel drive characteristics of the GS versus the front-wheel drive of the Avalon. Combined with the available 4-wheel steering, there is no doubt the GS is the better driver's car than the Avalon. And because it is a Lexus, it will have equipment not available on a Toyota, like a Mark Levinson sound system, power rear sunshade, etc. But the pricing reflects all this. The Avalon starts at $35K and tops out at $43k (give or take), whereas the GS starts at $51k and $85k for the GS F. It comes down to whether you feel the rear-wheel drive characteristics and Lexus equipment are worth the $10k+ difference in price. If comparing a new Avalon versus a used Lexus, that again comes down to personal preference and perception of value. I would personally go with new, because I tend to go above and beyond routine maintenance and keep the vehicle a long time, but everyone is different.
A smaller difference is the fit-and-finish: all GS' are made in Japan, whereas all Avalon's are made in Kentucky. I will show in a post soon all the little detail issues I'm having with my Avalon, from a waxy overspray in the trunk to sunroof headliner rattles to uneven panels to squeeky brake pedal noise etc. that I'm sure the GS doesn't have. They are not deal-breakers for me, but the fit-and-finish aren't on the same level as Japanese made Toyota's.