How Waymo tests its collision avoidance capabilities

Every driver will, inevitably, face unexpected hazards on the road, like other drivers running red lights or suddenly changing lanes. Autonomous vehicles (AVs) are no different, and AV developers have to find ways to prepare their autonomous drivers for as many unexpected events as possible. 

Waymo, the self-driving unit of Google-parent Alphabet, recently gave some insight into how it trains its Waymo Driver to avoid collisions on the road. The company recently published a paper detailing how it judges good collision avoidance performance, how it identifies the right set of scenarios to test and the testing tools it has developed to evaluate the Waymo Driver’s performance. 

Waymo is currently operating fully driverless robotaxi services in Chandler, Arizona, Downtown Phoenix and San Francisco, but before rolling out any of those services, the company tested its Driver extensively. To determine whether its Driver is ready, Waymo compares its performance against the performance of a reference model of a non-impaired human driver that always has eyes on the road, called NIEON for Non-Impaired with Eyes always On the conflict. 

NIEON is a model of a driver that surpasses the abilities of human drivers because it is always able to stay focused on what’s happening on the road. This means it creates a very high benchmark for the Waymo Driver to compete with, and the company has found that its Driver outperforms or demonstrates a comparable performance to NIEON.

Waymo found that the NIEON model could prevent 62% of crashes entirely, and reduce serious injury risk by 84%. The Waymo Driver, however, still did better, preventing 75% of collisions and reducing serious injury risk by 93%. 

Putting the Waymo Driver to the test

Waymo tests its Driver using three different methods: staging scenarios on closed tracks, using examples Waymo runs into during on-road testing and with fully synthetic simulations. Waymo’s real-world examples are constantly being updated with new scenarios the company runs into on the road. It uses fully synthetic simulations for situations that are too dangerous to stage, like for very fast-moving crashes, or for scenarios are too complicated to stage, like multi-lane intersections. 

Along with the millions of miles of driving data Waymo has gathered over years of testing, the company also uses human crash data, like police accident databases and crashes recorded by dash cams, and expert knowledge about its operation design domain, like geographic areas, driving conditions and road types where the Driver will operate, to decide what scenarios are the most important for it to test. 

Waymo has been gathering data for its scenario database since 2016, and it continues to add unique scenarios that it runs into on the roads to it. During its research, Waymo has found that the most common types of crashes are similar in any city, so its database can also help it to scale quickly in new cities. 

Waymo isn’t the only autonomous vehicle company to give insight into the safety of its robotaxis. Cruise recently released its safety report to give the public insights on what the company does to ensure its robotaxis are safe. The report details the approaches, tenets and processes that help keep Cruise vehicles safe on the road.