Would you buy a car which you drove out to the highway, set it on autopilot, and then took back control at the exit closest to your destination? Because that's how most commercial airlines fly these days. And more and more trains these days are operating driverless and guardless too (sometimes with an onboard backup driver, sometimes without), and whilst being on rails does remove some of the risk a fault in the system (either with the automation or the signalling) can still cause a potential catastrophe, yet I'd envisage most patrons couldn't notice the difference (apart from the train reliably stopping at a particular spot on the platform).
As with most technological advances these days, the technology is being developed and introduced before the infrastructure can properly handle it on a full scale. And as usual, there needs to be an agreement amongst manufacturers to work towards a common, standardised set of guidelines for all autonomous vehicles to use so they can integrate efficiently, effectively and safely, otherwise the thing will always remain a potential clusterfuck.
Hundreds of autonomous cars sharing a small patch of road together need to be talking to one another and coordinating their movements, or even platooning together in to car-trains (much like a cycling peloton) - if it was "every car for itself" it would be chaos. And the road needs to be able to support and handle that coordination - advanced lane-markings, wireless transmitting beacons to supplement signs, even built-in routing and navigation guides to reduce the reliance on GPS (especially in cities). And above all that, the main thing that separates cars from planes and trains - the ability to handle a traditional manually-controlled car in the thick of it, doing dumb-people things.
