NVIDIA offers ROS developers generative AI tools plus simulation, perception workflows

Today at ROSCon in Odense, Denmark, NVIDIA Corp. and its partners announced generative artificial intelligence tools, simulation, and perception AI workflows for developers using the Robot Operating System, or ROS.

Among the announcements were NVIDIA Isaac ROS and NVIDIA Isaac Sim technologies and developer guides, as well as NVIDIA AI-enabled generative AI ROS nodes for developers deploying to NVIDIA Jetson. Physically based simulation in Isaac Sim helps ROS developers create and validate AI models prior to deployment, according to the company.

Generative AI enables robots to perceive and understand the context of their surroundings, communicate naturally with humans, and make adaptive decisions autonomously, NVIDIA explained.

ROS community gets generative AI tools

Built on ROS 2, ReMEmbR uses generative AI to enhance robotic reasoning and action. It combines large language models (LLMs), vision language models (VLMs) and retrieval-augmented generation to allow robots to build and query long-term semantic memories and improve their ability to navigate and interact with their environments.

The speech-recognition capability is powered by the WhisperTRT ROS 2 node. This node uses NVIDIA TensorRT to optimize OpenAI’s Whisper model to enable low-latency inference on Jetson, resulting in responsive human-robot interaction.

The ROS 2 robots with voice control project uses the NVIDIA Riva ASR-TTS service to make robots understand and respond to spoken commands. The NASA Jet Propulsion Laboratory independently demonstrated ROSA, an AI-powered agent for ROS, operating on its Nebula-SPOT robot in Isaac Sim (see video below). 

At ROSCon, Canonical is demonstrating NanoOWL, a zero-shot object-detection model running on the Jetson Orin Nano system-on-module (SoM). It allows robots to identify a broad range of objects in real time, without relying on predefined categories.

Developers can get started today with ROS 2 Nodes for Generative AI, which brings NVIDIA Jetson-optimized LLMs and VLMs to enhance robot capabilities.

A ‘sim-first’ approach can enhance ROS workflows

Simulation is critical to safely test and validate AI-enabled robots before deployment said NVIDIA. Isaac Sim, a robotics simulation reference application built on OpenUSD and RTX, provides developers a virtual environment to test robots by connecting to their ROS packages.

A new “Beginner’s Guide to ROS 2 Workflows With Isaac Sim” is now available. It illustrates the end-to-end workflow for robot simulation and testing.

Foxglove, a member of the NVIDIA Inception program for startups, demonstrated an integration that helps developers visualize and debug simulation data in real time using its custom extension built on Isaac Sim.

Isaac ROS 3.2 adds capabilities

Isaac ROS, built on the open-source ROS 2 software framework, is a suite of accelerated computing packages and AI models for robotics development. The upcoming 3.2 release enhances robot perception, manipulation and environment mapping said NVIDIA.

Key improvements to Isaac Manipulator include new reference workflows that integrate FoundationPose and cuMotion to accelerate development of pick-and-place and object-following pipelines in robotics.

Another is to Isaac Perceptor, which features a new visual SLAM (simultaneous localization and mapping) reference workflow, enhanced multi-camera detection, and 3D reconstruction to improve an autonomous mobile robot’s (AMR) environmental awareness and performance in dynamic settings like warehouses.

Robotics developers adopt NVIDIA Isaac

NVIDIA listed robotics companies that are integrating Isaac ROS accelerated libraries and AI models into their platforms:

• Universal Robots, a Teradyne Robotics company, launched a new AI Accelerator toolkit to enable the development of AI-powered collaborative robot applications.
• Miso Robotics is using Isaac ROS to speed up its AI-powered robotic Flippy Fry Station and drive advances in efficiency and accuracy in food service automation.
• Wheel.me is partnering with RGo Robotics and NVIDIA to create a production-ready AMR using Isaac Perceptor.
• Main Street Autonomy is using Isaac Perceptor to streamline sensor calibration.
• Orbbec announced its Perceptor Developer Kit, an out-of-the-box AMR system for Isaac Perceptor.
• LIPS Corp. introduced a multi-camera perception devkit for improved AMR navigation.
• Canonical highlighted a certified Ubuntu environment for ROS developers, offering long-term support.

NVIDIA partners to present

ROS community members and partners, including Canonical, Ekumen, Foxglove, Intrinsic, and Teradyne Robotics, will present in workshops, talks, booth demonstrations, and sessions, including:

• “Nav2 User Meetup” Birds of a Feather session with Steve Macenski from Open Navigation LLC
• “ROS in Large-Scale Factory Automation” with Michael Gentner from BMW AG and Carsten Braunroth from Siemens AG
• “Integrating AI in Robot Manipulation Workflows” Birds of a Feather session by NVIDIA
• “Accelerating Robot Learning at Scale in Simulation” Birds of a Feather session by NVIDIA
• “On Use of Nav2 Docking” with Open Navigation’s Macenski

In addition, Teradyne Robotics and NVIDIA are co-hosting a lunch and evening reception at ROSCon today. The Open Source Robotics Foundation (OSRF) organizes the event. NVIDIA is a supporter of Open Robotics, the umbrella organization for OSRF and all its initiatives.

About the author

Kalyan Vadrevu is a product marketing manager at NVIDIA, where he focuses on the ROS developer ecosystem, Isaac ROS, and Isaac Manipulator. Before joining NVIDIA, Kalyan worked in developer relations and software development kit (SDK) marketing at Microsoft and Nokia. He has an MBA in marketing from Indiana University Bloomington.