I had to change my mirror and it is was very easy.
datnguyen24 is right about the black triangle. However, you will will have to remove the door panel to disconnect and reconnect the mirror if it is a power mirror. I don't see how you could pull the panel back and somehow disconnect it because there is a sheet of metal on top of the door panel that slides into the frame.
1) Pop off the black triangle.
2) Remove the door panel.
3) Disconnect the old mirror.
4) Unbolt the old mirror.
5) Do the steps with reverse with new mirror.
There is a DIY thread with pictures showing how to take the door panel off. Once you undo all the bolts the bottom and the sides of the panel pops off rather easily. To pop off the top of the panel you have to push upwards on the panel so that metal sheet slides out.
http://www.toyotanation.com/forum/t145402.html