In most Toyota power window systems, the windows receive power from two sources. One is controlled by drivers switch. Two wires run from the drivers switch console through the passenger switch and to the motor. Moving the drivers control switch one-way or the other changes the DC polarity to the motor. The motor will then run CW or CCW depending on DC polarity. NOTE: in this setup power to the motor goes through the two wires, the motor does not use the cars frame as an electrical ground.
The other power source is ONE wire going from the drivers console (always hot) to the passenger switch. This allows the passenger switch to control the window independently from the drivers console control switch.
When the passenger’s side switch is moved, the hot wire power is directed to one or the other wire of the motor, simultaneously the DC power return is directed through one of the wires that the drivers console use to control the motor. Thus the circuit is completed.
You will need to determine if power is coming from both drivers console wires, and is changing polarity when the switch is moved one-way or the other. And then determine if the hot wire has power and this power is being returned through one of the two wires discussed above.
Suggest you download the power window info for the Camry at the link below and look at the wiring diagrams. Should be close to what you have.
http://oregonstate.edu/~tongt/camry