It's a fine line to split, but there is a difference between "factory-installed" and "port-installed". The terminology comes from decades ago when the cars were made in Japan and some accessories were added after they got unloaded in North America. Aside from where the accessory/feature is installed, another factor is the region of the country the car is being shipped to. Again, decades ago the US was divided into a number of different regions and Toyota contracted with local companies to be regional distributors, and it was up to those distributors which accessories may be added to any given car. Fast forward to today, and Toyota owns all of those regional distributors except two (Southeast Toyota and Gulf States Toyota); SET and GST have a much larger menu of accessories and add-ons to choose from.
The window sticker is printed during distribution, not at the factory. That's why the window sticker includes those accessories, along with the name/address of the dealer it's being shipped to.
The trailer hitch receiver is a "port installed" option, along with the other things mentioned such as roof rack rails, floor mats, wheel locks, and so on.
Most port-installed options can be installed by a dealer as well, as the parts for those options are usually available through the region's parts distribution system.
So why is so hard to find ones on the lot with the hitch pre-installed? Because your regional distributor isn't ordering them that way. It must not be a popular item in your region.