Very similar problem and here is what I did.
We had 04 Sienna LE AWD and other car (not AWD) at home. Right after someone rear ended and totaled our Sienna, we planned to get the 2011 XLE AWD. Worked with some local dealers back and forth, test drive it many times, the more we test it, the more we DISLIKE it compare to our older Sienna. We really couldn't pass the down graded interior, too cheap even compared to our LE.
We also have 2 boys + my parents so the 2011 AWD is the only way for us since it has a better 3rd row access (without the 2nd row step over middle seat base). Not to mention we really want to have at least 1 car in the family that has AWD.
After going back and forth for a few weeks, then months.. we decided no 2011 Sienna. Instead we looked for 2004-2010 Sienna. Found a limited with low miles but dealer wouldn't give me a deal so I went shopping around the area and found a 07 Highlander. The same car we had before just a little newer on year (had 01 HL many years ago).
I still remembered we had the 01 for a couple years, sold it with only $$500 less then what we paid for. With the 07, I got a good deal on it and I am pretty sure it will hold better resale then the Sienna. So with the HL AWD V6, we now has a AWD at home and in a couple years, if Toyota improve the quality on the current Sienna, we will go back.
In your case, since you only have 2 kids, Sienna is not necessary. I have no trouble with our kids on the 07 so you should be ok on your 2010. I am pretty sure your HL will hold up the value better, that's the key.
Stop thinking about it for days, weeks, I am sure your buying instant will decrease and you will probably know for sure keeping the HL is a better deal