Imagine if you could pick between Uber drivers based on their driving experience. Would you hire an experienced driver who has logged hundreds of thousands of road miles or one who has driven just a few hundred miles? I’ll bet you’d go with the experienced driver.
Now apply the same question to driverless cars. How would you pick? The same logic applies: Go with experience.
By the miles-driven heuristic, recent reports released by the California Department of Motor Vehicles show that Waymo (the new Alphabet spinout previously known as Google's Self-Driving Car program) is running laps around its competitors. As with human drivers, experience matters for driverless capabilities. That’s because the deep learning AI techniques used to train driverless cars depend on data—especially data that illuminates rare and dangerous “edge cases.” The more training data, the more confidence you can have in the results.
See also: How to Picture the Future of Driverless
In 2016, Waymo logged more than 635,000 miles while testing its autonomous vehicles on California’s public roads compared to just over 20,000 for all its competitors combined.
As the W. Edwards Deming principle that is popular in Silicon Valley goes, “In God we trust, all others bring data.” The data shows that Waymo is not only 615,000 miles ahead of its competitors but that those competitors are still neophytes when it comes to proving their technology on real roads and interacting with unpredictable elements such as infrastructure, traffic and human drivers.
Now, there are lots of ways to cut the data and therefore a lot of provisos to the simple test-miles-driven heuristic.
Waymo also leads the others in terms of fewer “disengagements,” which refers to when human test drivers have to retake control from the driverless software. Waymo's test drivers had to disengage 124 times, or about once very 5,000 miles.
Other companies were all over the map in terms of their disengagements. BMW had one disengagement during 638 total miles of testing. Tesla had 182 disengagements in 550 miles. Mercedes-Benz had 336 disengagements over 673 miles. Fewer miles might mean fewer edge cases were encountered, or it might mean that those companies tested particularly difficult scenarios. But, low total miles driven casts doubt on the readiness of any system for operating on public roads. Until other contenders ramp up their total miles by a factor or 1,000 or more, their disengagement statistics are not statistically relevant.
Tesla fans could rightly point to the more than two hundred million miles that Tesla owners have logged under Tesla’s Autopilot feature. Those miles are not considered here. (Autopilot is not defined as autonomous under California law, so Tesla is not required to report disengagements to the California DMV.) But, no doubt, all those miles means that Tesla’s Autopilot software is probably very well trained for highway driving.
What do those highway miles tell us about Tesla’s ability to handle city streets, which are more complex for driverless cars? Not much, but the 550 miles that Tesla did spend on public road autonomous testing speaks volumes about its dearth of experiential learning on city streets. (Ed Niedermeyer, an industry analyst, recently argued that most of Tesla's 550 miles were probably logged while filming one marketing video.)
See also: Novel Solution for Driverless Risk
It should also be noted that the reported data applies only to California; it does not account for testing in other active driverless hubs—such as Waymo’s test cars in Austin, TX, Uber’s driverless pilots in Pittsburgh or nuTonomy’s testing in Singapore (just to name a few). It is safe to guess, however, that a significant percentage of all autonomous testing has been logged in California.
Notably missing from the reports to the California DMV are all other Big Auto makers and suppliers—and other players cited or rumored as driverless contenders, like Apple and Baidu. They might well be learning to drive on private test tracks or outside of California. But, until they bring data about their performance after significant miles on public roads, don't trust the press releases or rumors about their capabilities.
Waymo’s deep experience in California does not guarantee its victory. Can it stay ahead as others accelerate? That remains to be seen, but it is clear from the California DMV reports that Waymo is way ahead on the driverless learning curve.
As the W. Edwards Deming principle that is popular in Silicon Valley goes, “In God we trust, all others bring data.” The data shows that Waymo is not only 615,000 miles ahead of its competitors but that those competitors are still neophytes when it comes to proving their technology on real roads and interacting with unpredictable elements such as infrastructure, traffic and human drivers.
Now, there are lots of ways to cut the data and therefore a lot of provisos to the simple test-miles-driven heuristic.
Waymo also leads the others in terms of fewer “disengagements,” which refers to when human test drivers have to retake control from the driverless software. Waymo's test drivers had to disengage 124 times, or about once very 5,000 miles.
Other companies were all over the map in terms of their disengagements. BMW had one disengagement during 638 total miles of testing. Tesla had 182 disengagements in 550 miles. Mercedes-Benz had 336 disengagements over 673 miles. Fewer miles might mean fewer edge cases were encountered, or it might mean that those companies tested particularly difficult scenarios. But, low total miles driven casts doubt on the readiness of any system for operating on public roads. Until other contenders ramp up their total miles by a factor or 1,000 or more, their disengagement statistics are not statistically relevant.
Tesla fans could rightly point to the more than two hundred million miles that Tesla owners have logged under Tesla’s Autopilot feature. Those miles are not considered here. (Autopilot is not defined as autonomous under California law, so Tesla is not required to report disengagements to the California DMV.) But, no doubt, all those miles means that Tesla’s Autopilot software is probably very well trained for highway driving.
What do those highway miles tell us about Tesla’s ability to handle city streets, which are more complex for driverless cars? Not much, but the 550 miles that Tesla did spend on public road autonomous testing speaks volumes about its dearth of experiential learning on city streets. (Ed Niedermeyer, an industry analyst, recently argued that most of Tesla's 550 miles were probably logged while filming one marketing video.)
See also: Novel Solution for Driverless Risk
It should also be noted that the reported data applies only to California; it does not account for testing in other active driverless hubs—such as Waymo’s test cars in Austin, TX, Uber’s driverless pilots in Pittsburgh or nuTonomy’s testing in Singapore (just to name a few). It is safe to guess, however, that a significant percentage of all autonomous testing has been logged in California.
Notably missing from the reports to the California DMV are all other Big Auto makers and suppliers—and other players cited or rumored as driverless contenders, like Apple and Baidu. They might well be learning to drive on private test tracks or outside of California. But, until they bring data about their performance after significant miles on public roads, don't trust the press releases or rumors about their capabilities.
Waymo’s deep experience in California does not guarantee its victory. Can it stay ahead as others accelerate? That remains to be seen, but it is clear from the California DMV reports that Waymo is way ahead on the driverless learning curve.
