Inside Waymo’s safety benchmarks for robotaxis

While autonomous vehicles are being deployed in cities across the U.S., like Austin, Las Vegas, Miami, Phoenix and San Francisco, members of the public still have many questions about how safe the vehicles are and they compare to human drivers.

Currently, there aren’t any universally accepted approaches for evaluating the safety of autonomous driving systems, so Waymo, the self-driving unit of Google’s parent company Alphabet, released its Response Time paper and Collision Avoidance Benchmarking paper to provide more clarity on the safety benchmarks it holds for its vehicles. The papers outline how the Waymo Driver compares to average human drivers and better than average drivers.

To compare human drivers to the Waymo Driver, the company first needed to determine how well human drivers are able to avoid collisions. Typically, human response timing has been evaluated during controlled experiments in artificial environments. There, subjects are instructed to respond to well-defined stimulus signals, like a loud sound or a brake light onset. 

While this method can be effective, it misses a few key aspects of driving in the real world, according to Waymo. Human drivers don’t always behave based on well-defined instructions, and simulated scenarios don’t account for the urgency of an impending crash on the road. It can also be difficult for researchers to determine when to “start the clock” to gauge response time, especially in slowly unfolding scenarios. 

As a result, Waymo said these simulated models overestimate a human driver’s response time, predicting that humans will respond later, in fast-moving situations. It also underestimates a driver’s response time in slow-developing situations, where humans can struggle to respond with urgency. 

Waymo’s framework aims to be more accurate than typical models by using data from naturalistic driving studies. The framework it uses is based on two key ideas. The first is that humans decide to brake or steer to avoid a crash because something in their surroundings surprised them. The onset of this surprise is when Waymo starts the clock.

The second is that human responses will change depending on the evolving situation, so it can’t be represented by a single fixed response time that can be applied to all scenarios. 

Waymo used this framework and an evasive maneuver model to create an internal benchmark for collision avoidance behavior that exceeds typical human drivers. This reference model accounts for a human driver that is always paying attention, never impaired, and responds to every situation as quickly as possible. In Waymo’s words, it’s a driver that is Non-Impaired, with Eyes always On the conflict (NIEON).

To test how NIEON and the Waymo Driver perform against human drivers, Waymo compared the simulated performance of the Waymo driver to human drivers involved in a series of fatal crashes that happened between 2008 and 2017 in Chandler, Arizona. The company found that the Waymo Driver completely avoided or mitigated 100% of crashes, except for those where the driver was struck from behind. 

Waymo then tested how NIEON would behave in these crashes, as NIEON would present a higher benchmark than average human drivers. They found that the NIEON model could prevent 62% of crashes entirely, and reduce serious injury risk by 84%. The Waymo Driver, however, still did better, preventing 75% of collisions and reducing serious injury risk by 93%. 

Waymo is delivering a keynote at RoboBusiness, which runs Oct. 19-20 in Santa Clara and is produced by WTWH Media, the parent company of The Robot Report. Allison Thackston, technical lead & manager at Waymo, will discuss on Oct. 19 from 9:30-10:15 AM the technical challenges of developing autonomous vehicles. She will also provide updates on Waymo’s ongoing deployments and work. Prior to Waymo, Thackston worked at Nuro, Toyota Research Institute and Oceaneering. She’s also a founding member of the ROS2 Technical Steering Committee.