The Robot Report

  • Home
  • News
  • Technologies
    • Batteries / Power Supplies
    • Cameras / Imaging / Vision
    • Controllers
    • End Effectors
    • Microprocessors / SoCs
    • Motion Control
    • Sensors
    • Soft Robotics
    • Software / Simulation
  • Development
    • Artificial Intelligence
    • Human Robot Interaction / Haptics
    • Mobility / Navigation
    • Research
  • Robots
    • AGVs
    • AMRs
    • Consumer
    • Collaborative Robots
    • Drones
    • Humanoids
    • Industrial
    • Self-Driving Vehicles
    • Unmanned Maritime Systems
  • Business
    • Financial
      • Investments
      • Mergers & Acquisitions
      • Earnings
    • Markets
      • Agriculture
      • Healthcare
      • Logistics
      • Manufacturing
      • Mining
      • Security
    • RBR50
      • RBR50 Winners 2025
      • RBR50 Winners 2024
      • RBR50 Winners 2023
      • RBR50 Winners 2022
      • RBR50 Winners 2021
  • Resources
    • Automated Warehouse Research Reports
    • Digital Issues
    • eBooks
    • Publications
      • Automated Warehouse
      • Collaborative Robotics Trends
    • Search Robotics Database
    • Videos
    • Webinars / Digital Events
  • Events
    • RoboBusiness
    • Robotics Summit & Expo
    • DeviceTalks
    • R&D 100
    • Robotics Weeks
  • Podcast
    • Episodes
  • Advertise
  • Subscribe

How Zoox robotaxis make predictions while on the road

By Brianna Wessling | August 17, 2022

zoox taxi

Zoox’s robotaxis have sensors placed high on the vehicle’s four corners that give it a 360º view of its surroundings. | Source: Zoox

Amazon acquired Zoox back in January 2020 for what reports suggested was around $1.2 billion. Since then, the company has revealed its Zoox vehicle, a rectangular passenger-focused vehicle with no driver’s seat or steering wheel and expanded testing facilities, but news from the company has been otherwise quiet. 

Amazon recently demonstrated how the Zoox vehicle can predict its surroundings up to eight seconds in the future. These seconds allow the vehicle to react and make prudent and safe driving decisions.

Zoox’s artificial intelligence (AI) stack is at the heart of the vehicle’s ability to predict these outcomes. To accomplish this, the stack employs three broad processes: perception, prediction, and planning.

Predicting the future

Zoox’s AI stack starts with its perception stage, where the vehicle takes in everything in its surroundings and how each thing is moving. 

The perception phase begins with high-resolution data that Zoox’s team gathers from the vehicle’s sensors. Zoox is equipped with a variety of sensors, from visual cameras to LiDAR, radar, and longwave infrared cameras. These sensors are placed at the high four corners of the vehicle, giving Zoox an overlapping, 360º view of the car’s surroundings for over 100 m. 

The robotaxi combines this data with an already provided, detailed semantic map of its environment called the Zoox Road Network (ZRN). The ZRN has information about local infrastructure, road rules, speed limits, intersection layouts, location of traffic symbols and more. 

The perception AI then identifies and classifies surrounding cars, pedestrians and cyclists, which it calls “agents.” The AI tracks each of these agent’s velocities and trajectories. It then boils down this data to its essentials, making it into a 2D image optimized for machine learning to understand. 

This image is presented to a convolutional neural network, which decides what items in the image matter to the vehicle. The image includes around 60 channels of semantic information about all of the agents in it. 

With this information, the machine learning system creates a probability distribution of potential trajectories for each dynamic agent in the vehicle’s surroundings. The machine learning system considers the trajectory of all agents, as well as how cars are expected to move on a given street, what traffic lights are doing, the workings of crosswalks and more. 

The system’s resulting predictions are usually around eight seconds into the future, and are recalculated every tenth of a second with new information from the perception system. 

Weighted predictions are given to the final stage of the process, the planner phase. The planner is the car’s executive decision-making. It takes predictions from the previous phase and uses it to decide how the Zoox vehicle will move. 

Constantly improving predictions

While Zoox’s AI stack has millions of miles of sensor data collected by the company’s test fleet to train from, the team is still constantly trying to improve its accuracy. 

Right now, the team is working to leverage a graph neural network (GNN) approach to improve the stack’s prediction capabilities. A GNN would enable the vehicle to understand the relationships between different agents around it and within itself, as well as how those relationships will change over time.

The team is also working to more deeply integrate the prediction and planning stages of the process to create a feedback loop. This would allow the prediction and planner systems to interact by allowing the planner system to ask the prediction system how agents might react to certain behaviors before carrying out decisions.

About The Author

Brianna Wessling

Brianna Wessling is an Associate Editor, Robotics, WTWH Media. She joined WTWH Media in November 2021, after graduating from the University of Kansas with degrees in Journalism and English. She covers a wide range of robotics topics, but specializes in women in robotics, autonomous vehicles, and space robotics.

She can be reached at bwessling@wtwhmedia.com

Tell Us What You Think! Cancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Related Articles Read More >

Artedrone's Sasha autonomous mechanical thrombectomy system.
Microrobot system is designed to float inside stroke patient for autonomous thrombectomy
Mbodi AI is working with partners such as ABB.
Mbodi AI launches on Y Combinator, developing embodied AI for industrial robots
Photoneo MotionCam 3D Color (Blue) enables scanning from greater distances as in this palletizing application, says Photoneo.
Photoneo releases MotionCam-3D Color (Blue) to improve robot perception
Kawasaki had one of the largest robots at Automate 2025.
10 robotics trends spotted at Automate 2025

RBR50 Innovation Awards

“rr
EXPAND YOUR KNOWLEDGE AND STAY CONNECTED
Get the latest info on technologies, tools and strategies for Robotics Professionals.
The Robot Report Listing Database

Latest Episode of The Robot Report Podcast

Automated Warehouse Research Reports

Sponsored Content

  • Sager Electronics and its partners, logos shown here, will exhibit at the 2025 Robotics Summit & Expo. Sager Electronics to exhibit at the Robotics Summit & Expo
  • The Shift in Robotics: How Visual Perception is Separating Winners from the Pack
  • An AutoStore automated storage and retrieval grid. Webinar to provide automated storage and retrieval adoption advice
  • Smaller, tougher devices for evolving demands
  • Modular motors and gearboxes make product development simple
The Robot Report
  • Mobile Robot Guide
  • Collaborative Robotics Trends
  • Field Robotics Forum
  • Healthcare Robotics Engineering Forum
  • RoboBusiness Event
  • Robotics Summit & Expo
  • About The Robot Report
  • Subscribe
  • Contact Us

Copyright © 2025 WTWH Media LLC. All Rights Reserved. The material on this site may not be reproduced, distributed, transmitted, cached or otherwise used, except with the prior written permission of WTWH Media
Privacy Policy | Advertising | About Us

Search The Robot Report

  • Home
  • News
  • Technologies
    • Batteries / Power Supplies
    • Cameras / Imaging / Vision
    • Controllers
    • End Effectors
    • Microprocessors / SoCs
    • Motion Control
    • Sensors
    • Soft Robotics
    • Software / Simulation
  • Development
    • Artificial Intelligence
    • Human Robot Interaction / Haptics
    • Mobility / Navigation
    • Research
  • Robots
    • AGVs
    • AMRs
    • Consumer
    • Collaborative Robots
    • Drones
    • Humanoids
    • Industrial
    • Self-Driving Vehicles
    • Unmanned Maritime Systems
  • Business
    • Financial
      • Investments
      • Mergers & Acquisitions
      • Earnings
    • Markets
      • Agriculture
      • Healthcare
      • Logistics
      • Manufacturing
      • Mining
      • Security
    • RBR50
      • RBR50 Winners 2025
      • RBR50 Winners 2024
      • RBR50 Winners 2023
      • RBR50 Winners 2022
      • RBR50 Winners 2021
  • Resources
    • Automated Warehouse Research Reports
    • Digital Issues
    • eBooks
    • Publications
      • Automated Warehouse
      • Collaborative Robotics Trends
    • Search Robotics Database
    • Videos
    • Webinars / Digital Events
  • Events
    • RoboBusiness
    • Robotics Summit & Expo
    • DeviceTalks
    • R&D 100
    • Robotics Weeks
  • Podcast
    • Episodes
  • Advertise
  • Subscribe