That is certainly a strong possibility for a few of the situations I've noticed the A/C being on unexpectedly -- I usually start out in the morning with air blowing up onto the windshield so I get moving air yet it won't be directly in my face and this meets your criteria.
But I have also noticed the A/C seemingly always being on later in the day when using the main dash vents, so the defrost equals A/C on scenario wouldn't apply there.
I can't test that again now though as everything is all put back together and working properly -- and it may be a case of fixing a problem by taking everything apart and doing nothing but putting it all back together again.
Yesterday I was chatting with the Toyota Parts Guy at the local dealership about my problem; he was nice enough to ask one of the mechanics if he had any idea as to what to check. He suggested I see if the compressor clutch was somehow not disengaging, an easy thing to check. This seemed like a hot tip so I went right home to try it out.
The main belt goes around the outer wheel of the A/C compressor. This wheel spins around the inner wheel which is actually connected to the compressor, only turning the inner wheel when the clutches engage. So you can reach in (with the engine off!) and turn the inner wheel by hand and the only resistance you will feel is that of the compressor itself. If you cannot turn the inner wheel at all (or with great difficulty), that means it is locked to the outer wheel (clutches engaged) and no wheels will turn as the belt isn't turning. This would explain a scenario where the A/C was always on.
In my case, the inner wheel turned freely. I decided to check further and see what would happen with the inner wheel under the normal, engine running scenario.
With the engine off, I put a dot of white paint on the rim of the inner wheel. I put the inside center console heater control module back into the dash just enough to hook everything up electrically.
Started up the engine (with A/C button disengaged and fan off), looked into the engine compartment and could clearly see the white dot on the inner compressor wheel just sitting there motionless, with the outer wheel spinning around the inner wheel as the belt rotated it. Definite confirmation that there was no mechanical problem with the compressor wheels being locked together, causing the A/C to always run.
It suddenly occurs to me that if the belt is turning the outer wheel but not turning the inner wheel, that means that the compressor isn't running and there is no A/C. Doesn't this mean that the A/C is now no longer running all the time?
Push the A/C button on the dash and the compressor clutch engages and A/C mode is working. Push the A/C button on the dash again to turn the A/C off and danged if the A/C doesn't turn off just like it is supposed to. Button in and A/C on. Button out and A/C off. In/On, Out/Off. In/On, Out/Off.
I'll be danged to the lukewarms pits of heck. My A/C is working properly now.
I figure there are only two possible options -- either I was a complete doofus and completely mis-interpreted all my testing prior to spending hours figuring out how to take everything apart and then spending hours actually taking everything apart
taking the A/C control panel apart and/or pulling various fuses and relays and putting them back in somehow remedied a bad connection
I'm going with the latter but won't bet the farm it wasn't the former.
For future reference to folks who are interested, on my 2000 Camry CE the two lamps that light up the heater control panel (fan speed, air temp, vent selector knobs) are NOT #74 bulbs as most people report -- they are JKL EWG103 and are Toyota Part 90069-98004. You can buy these bulbs online for $0.73 each plus shipping or pay $4.00+ each at your local dealership and they do NOT come with the little green condoms even at that exhorbitant price.