I'm surprised you ever use the rear window defroester in NM. It must get colder at night than I imagined. This same 50A fuse also protects the side mirror heaters (if your Avalon has them). If you could determine if they (the mirror heaters) are working, the fuse and relay could be eliminated as a cause, leaving a broken wire at the window grid termination as more likely.
If you have a volt-ohm meter and are comfortable using it, I can describe where to probe the relay socket (after pulling the relay) to determine if the fuse, wiring, and heater grid are OK.