The Robot Report

  • Home
  • News
  • Technologies
    • Batteries / Power Supplies
    • Cameras / Imaging / Vision
    • Controllers
    • End Effectors
    • Microprocessors / SoCs
    • Motion Control
    • Sensors
    • Soft Robotics
    • Software / Simulation
  • Development
    • Artificial Intelligence
    • Human Robot Interaction / Haptics
    • Mobility / Navigation
    • Research
  • Robots
    • AGVs
    • AMRs
    • Consumer
    • Collaborative Robots
    • Drones
    • Exoskeletons
    • Industrial
    • Self-Driving Vehicles
    • Unmanned Maritime Systems
  • Markets
    • Agriculture
    • Healthcare
    • Logistics
    • Manufacturing
    • Mining
    • Security
  • Financial
    • Investments
    • Mergers & Acquisitions
    • Earnings
  • Resources
    • Careers
    • COVID-19
    • Digital Issues
    • Publications
      • Collaborative Robotics Trends
      • Robotics Business Review
    • RBR50 Winners 2022
    • Search Robotics Database
    • Videos
    • Webinars / Digital Events
  • Events
    • RoboBusiness
    • Robotics Summit & Expo
    • Healthcare Robotics Engineering Forum
    • DeviceTalks
    • R&D 100
    • Robotics Weeks
  • Podcast
    • Episodes
    • Leave a voicemail

MIT CSAIL releases open-source simulator for autonomous vehicles

By Brianna Wessling | June 22, 2022

Listen to this article
Voiced by Amazon Polly

Researchers at MIT’s Computer Science and Artificial Intelligence Laboratory have released an open-source simulation engine that can construct photorealistic environments to train and test autonomous vehicles.

Teaching neural networks to drive vehicles autonomously requires a large amount of data. Much of this data can be difficult to secure in the real world using real vehicles. Researchers can’t simply crash a car to teach a neural network to not crash a car, so they rely on simulated environments for this kind of data. That’s where simulated training environments, like CSAIL’s VISTA 2.0 comes in. 

VISTA 2.0, an updated version of the team’s previous model VISTA, is a data-driven simulation environment that was photorealistically rendered from real-world data. It’s able to simulate complex sensor types and interactive scenarios and intersections at scale. 

“Today, only companies have software like the type of simulation environments and capabilities of VISTA 2.0, and this software is proprietary. With this release, the research community will have access to a powerful new tool for accelerating the research and development of adaptive robust control for autonomous driving,” MIT Professor and CSAIL Director Daniela Rus, senior author on a paper about the research, said. 

VISTA 2.0’s photorealistic environment reflects a recent trend in the autonomous vehicle industry. Developers are moving away from using human-designed simulation environments and towards using ones built from real-world data. 

These environments are appealing because they allow for direct transfers to reality. However it can be difficult to synthesize the richness and complexity of all the sensors autonomous vehicles need. For example, to replicate LiDAR in these environments, researchers essentially need to generate new 3D point clouds with millions of points using only a sparse view of the world. 

To get around this, the MIT team projected data collected from the car into a 3D space coming from the LiDAR data. They then let a new virtual vehicle drive around locally from where that original vehicle was, and, with the help of neural networks, projected sensory information back into the frame of view of the new virtual vehicle. 

csail simulation

VISTA 2.0, can open-source simulation engine from MIT CSAIL, that can simulate environments for training self driving cars. | Source: MIT CSAIL

The team also simulated, in real-time, event-based cameras, which operate at greater speeds than thousands of events per second. With all of these sensors simulated, you’re able to move vehicles around in the simulation, simulate different types of events, and drop in brand new vehicles not part of the original data. 

“This is a massive jump in capabilities of data-driven simulation for autonomous vehicles, as well as the increase of scale and ability to handle greater driving complexity,” Alexander Amini, CSAIL PhD student and co-lead author on two new papers, together with fellow PhD student Tsun-Hsuan Wang, said. “VISTA 2.0 demonstrates the ability to simulate sensor data far beyond 2D RGB cameras, but also extremely high dimensional 3D lidars with millions of points, irregularly timed event-based cameras, and even interactive and dynamic scenarios with other vehicles as well.” 

MIT’s team took a full-scale car out to test VISTA 2.0 in Devens, Massachusetts. The team saw an immediate transferability of results, with both failures and successes. Moving forward, CSAIL hopes to allow the neural network to understand and respond to gestures from other drivers, like a wave, nod or blinker switch of acknowledgement. 

Amini and Wang wrote the paper alongside Zhijian Liu, MIT CSAIL PhD student; Igor Gilitschenski, assistant professor in computer science at the University of Toronto; Wilko Schwarting, AI research scientist and MIT CSAIL PhD ’20; Song Han, associate professor at MIT’s Department of Electrical Engineering and Computer Science; Sertac Karaman, associate professor of aeronautics and astronautics at MIT; and Daniela Rus, MIT professor and CSAIL director.

About The Author

Brianna Wessling

Brianna Wessling is an Associate Editor, Robotics, WTWH Media. She joined WTWH Media in November 2021, and is a recent graduate from the University of Kansas. She can be reached at [email protected]

Tell Us What You Think! Cancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Related Articles Read More >

A3 robots
Robot sales hit record high in first quarter of 2022
tiny robot on penny
Researchers create walking robot half a millimeter wide
CMU ATV
Carnegie Mellon researchers gather data to train self-driving ATVs
DeepMind’s open-source version of MuJoCo available on GitHub

2021 Robotics Handbook

The Robot Report Listing Database

Latest Robotics News

Robot Report Podcast

Anders Beck introduces the UR20; California bans autonomous tractors
See More >

Sponsored Content

  • Magnetic encoders support the stabilization control of a self-balancing two-wheeled robotic vehicle
  • How to best choose your AGV’s Wheel Drive provider
  • Meet Trey, the autonomous trailer (un)loading forklift
  • Kinova Robotics launches Link 6, the first Canadian industrial collaborative robot
  • Torque sensors help make human/robot collaborations safer for workers

RBR50 Innovation Awards

Leave us a voicemail

The Robot Report
  • Mobile Robot Guide
  • Collaborative Robotics Trends
  • Field Robotics Forum
  • Healthcare Robotics Engineering Forum
  • RoboBusiness Event
  • Robotics Business Review
  • Robotics Summit & Expo
  • About The Robot Report
  • Subscribe
  • Advertising
  • Contact Us

Copyright © 2022 WTWH Media LLC. All Rights Reserved. The material on this site may not be reproduced, distributed, transmitted, cached or otherwise used, except with the prior written permission of WTWH Media
Privacy Policy | Advertising | About Us

Search The Robot Report

  • Home
  • News
  • Technologies
    • Batteries / Power Supplies
    • Cameras / Imaging / Vision
    • Controllers
    • End Effectors
    • Microprocessors / SoCs
    • Motion Control
    • Sensors
    • Soft Robotics
    • Software / Simulation
  • Development
    • Artificial Intelligence
    • Human Robot Interaction / Haptics
    • Mobility / Navigation
    • Research
  • Robots
    • AGVs
    • AMRs
    • Consumer
    • Collaborative Robots
    • Drones
    • Exoskeletons
    • Industrial
    • Self-Driving Vehicles
    • Unmanned Maritime Systems
  • Markets
    • Agriculture
    • Healthcare
    • Logistics
    • Manufacturing
    • Mining
    • Security
  • Financial
    • Investments
    • Mergers & Acquisitions
    • Earnings
  • Resources
    • Careers
    • COVID-19
    • Digital Issues
    • Publications
      • Collaborative Robotics Trends
      • Robotics Business Review
    • RBR50 Winners 2022
    • Search Robotics Database
    • Videos
    • Webinars / Digital Events
  • Events
    • RoboBusiness
    • Robotics Summit & Expo
    • Healthcare Robotics Engineering Forum
    • DeviceTalks
    • R&D 100
    • Robotics Weeks
  • Podcast
    • Episodes
    • Leave a voicemail