I had the same problem on a 2004 CE Manual and found a solution as below. I cut and pasted from a quick search on TN but somewhere around here is a link to a pdf of the actual TSB. I had to use it to arm twist the boys at Downtown Toyota but they did eventually break down and replace the ECM under warranty. Works fine since.
"TSB2754 Idle Hunting in Sub-Freezing Ambient Temperatures
Some 2003 - 2004 model year
[COLOR=orange ! important][COLOR=orange ! important]Corolla[/COLOR][/COLOR] and Matrix vehicles may exhibit an "idle hunting" condition after the vehicle has been cold soaked in sub-freezing ambient air temperatures. During this condition the
[COLOR=orange ! important][COLOR=orange ! important]engine[/COLOR][/COLOR] idle will "hunt" or fluctuate instead of maintaining a constant fast idle speed. The Engine
[COLOR=orange ! important][COLOR=orange ! important]Control [COLOR=orange ! important]Module[/COLOR][/COLOR][/COLOR]
(ECM) logic has been modified to correct this condition.
Warranties on all 03/04 Corolla or Matrix for 96 months or 130000kms/80000ms with Manual Transmission"