There are a number of things that could be causing your starting issue. Need a bit more information:

Are you getting that ticking/clicking sound when you first tried to start it? That might indicate a weak battery, poor battery connections, or bad starter contacts.

Also possible, given that it seems to start right away after rowing through the gear shift, that your neutral safety switch could be faulty. That is part of a circuit that makes it permissible to start the car, don't want to start the car in gear - only park and neutral. Possible that it doesn't know exactly where the shift position is (bad electrical connection, lever arm on the side is damaged or not connected, etc.) - so it may not know what gear it is in - not allowing the starter to turn over.

 

Could it be that it wasn't in park all the way? I put my Venza in reverse and the camera didn't come on. I noticed the shift selector was just a little off.

 

Same thing on my '08. It was the neutral safety switch. Extended warranty replaced it to the tune of $400+.

 

As fgf001 said, most likely is a defective NEUTRAL/PARK switch. It detects the position of the shift lever and only allows the engine to be started in PARK or NEUTRAL, but not in DRIVE or REVERSE. It should be covered by the warranty. Even if you are little bit past the 3 yr / 36k miles warranty, I suspect that Toyota will cover at least part of the cost because it is a safety device. Don't wait any longer and get it fixed by the dealer. Otherwise, if the switch fails completely and you will end up stranded unable to start the engine.

 

Mine would not even start in neutral. It was a crap shoot if it would start or not. Since the replacement of the part no problems whatsoever.

Frankly it's bull$hit for a car with just over 21K miles. Glad I bought the warranty.