Thanks goog1986, that helped me a lot. I was trying to install front speakers so I had to remove the door panel.
Here are some things I wish I knew before I started (sorry for the huge pics)
The side pin/clip looks like this normally
When you want to take it out, depress the middle piece, like this
Then, when you want to put it in again, pull out the middle piece slightly, like this
The bezel that goes around the door handle is tough to get off, and I forgot to take a picture of it once I got it out. My suggestion would be to free the bottom part first, then the top, then slide rearwards. To free the bottom part, I pulled the handle back and stuck a flat-head screwdriver downwards (in that small gap between the handle base and the bezel). Everything else I think was simple, the bottom clips just pop out when you pull.
Thanks to you guys, it's awesome to have a forum with such great info!