Like #2! said, good chance the IAC is gummed up to the point it isn't working right in cold weather. See
DIY: Cleaning the Throttlebody (5S-FE - other engines similar) for a good writeup on the cleaning procedure.
Another possibility is that the Engine Coolant Temperature sensor (ECT) is reading too "warm". Is your Check Engine Light on? If that sensor fails completely, the CEL will be on, and the engine will default to "warmed up" operation -- not so good when the temperature is low. It's also possible for it to fail only partially, and you'd never see a CEL. The FSM at
www.camrystuff.com , Generation 3, 5S-FE section, starting on page 356 shows the test procedure, although you'll need a cheap-o multimeter to do the test.
If you've never cleaned the IAC and throttle body, I'd recommend doing that first -- it's easy to do, and after ~17 years it's guaranteed to be really gunked up.