There are two Torx screws (don't remember what size) in the conversation mirror compartment, then two clips on either side near the front that need to be disengaged, and then unplug the connectors.
There's a used one for sale on eBay (at LKQ) that includes some pictures:
http://www.ebay.com/itm/2004-Toyota-...sories&vxp=mtr
It looks like there's four more Torx screws, that hold the HomeLink/compass/thermometer module in from the back. You can also see where the clips are in one of the pictures. Good luck!