I would think it's more like wearing down the transmission and not the engine.
The "conventional" wisdom is to leave the OD button off, so the transmission does not go into OD for 1 second and then downshift again. This is quite commonly done on 4 speeds of all makes.
However, do not leave in L or 2 going uphill for too long. An additional brake band engages in those gears and the transmission can wear out more quickly. Use them only going downhill.
The interesting thing about newer 6-speed transmissions of all makes is that there is no such option (OD Off) any more. It's OD-On all the time or manual shift, which would be a hassle IMO.
Maybe this means might as well leave in OD all the time. :lol: (that said, to avoid frequent up/down shifts between 3/4 in some situations, I'd still turn OD off in 4-speed transmissions. Otherwise as you said, OD is left on).
Have you ever replaced the ATF and differential fluid? Maybe drop and clean the pan and replace the strainer? I'd use at least Walmart's SuperTech Mercon-V (suitable for Dexron III) or Castrol Import Multi-Vehicle ATF (also for D-III).
I live in the Foothills of the Blue Ridge Mountains, so it's kinda of hilly, but not San Francisco. I had a few people tell me that leaving it in OD could wear down the engine, but my dad said that most cars now have it built in and that Toyota just put that in there to make it look better. I am a little disseminated in only 20mpg, but I still like the car a lot.