It will be quite loud if you take out the carpet, door cards and rear seats. The 20-50lbs is worth it to me.
You should also consider removing the sound deadening on the floors. You can probably get at least 20 more pounds this way. Get some dry ice at the store, get it cold and hit it with a hammer... it'll crack off.
If you don't care about safety, you can also cut out most of your front and rear bumper beams. Remove cruise control (if you have it). Move the battery to the trunk and get a lighter one. Get a lighter driver's seat (racing seat).
This of course will all not really help you much if your suspension isn't already in good shape - springs, struts, swaybars, bushings, etc...
-Charlie