Heck, I had my Camry 14 months before I even found out you could unlock or lock all the doors instead of just the driver's door if you simply turn the key a second time within a few seconds. I never realized it, then after I hooked up my alarm I never have to use the keys in the doors.
But yes, the wires coming out of the door jamb that go into the door do break from so many years of opening and closing, so this could easily be your problem. I had that issue with my driver's window. It would only roll up or down when it wanted to! Sometimes it would move until I moved the door. I had to cut into the huge rubber tube the wires go through and indeed found a broken wire. Simple fix.
Here is how to test the switches (but I'm guessing it's a wire):
If you save this image and open it on your computer it will be big enough to read clearly.