The Robot Report

  • Home
  • News
  • Technologies
    • Batteries / Power Supplies
    • Cameras / Imaging / Vision
    • Controllers
    • End Effectors
    • Microprocessors / SoCs
    • Motion Control
    • Sensors
    • Soft Robotics
    • Software / Simulation
  • Development
    • Artificial Intelligence
    • Human Robot Interaction / Haptics
    • Mobility / Navigation
    • Research
  • Robots
    • AGVs
    • AMRs
    • Consumer
    • Collaborative Robots
    • Drones
    • Humanoids
    • Industrial
    • Self-Driving Vehicles
    • Unmanned Maritime Systems
  • Business
    • Financial
      • Investments
      • Mergers & Acquisitions
      • Earnings
    • Markets
      • Agriculture
      • Healthcare
      • Logistics
      • Manufacturing
      • Mining
      • Security
    • RBR50
      • RBR50 Winners 2025
      • RBR50 Winners 2024
      • RBR50 Winners 2023
      • RBR50 Winners 2022
      • RBR50 Winners 2021
  • Resources
    • Automated Warehouse Research Reports
    • Digital Issues
    • eBooks
    • Publications
      • Automated Warehouse
      • Collaborative Robotics Trends
    • Search Robotics Database
    • Videos
    • Webinars / Digital Events
  • Events
    • RoboBusiness
    • Robotics Summit & Expo
    • DeviceTalks
    • R&D 100
    • Robotics Weeks
  • Podcast
    • Episodes
  • Advertise
  • Subscribe

Toyota Research Institute teaches mobile manipulator with VR, simulation

By Eugene Demaitre | October 7, 2019


Robust mobile manipulation, which is starting to appear in supply chain operations, is a step toward service robots with more generalized capabilities. Toyota Research Institute last week posted a video (above) showing the state of its research on household robots.

Aging populations in Europe, East Asia, and North America pose challenges, according to a United Nations study cited by Toyota Research Institute (TRI). One of the goals of service robotics is to enable people to “age in place,” or remain in their homes with as much autonomy as possible and without scarce human aid.

Unlike other technology enterprises and automakers working on self-driving cars, Toyota has positioned itself as a “mobility company” as it looks to diversify its offerings beyond vehicles and into assisting households. As part of that effort, the company set an internal challenge of teaching robots to perform a variety of tasks in actual homes.

“Teaching robots to perform tasks is a key element of advancing fleet learning for enabling useful robot assistants in homes,” Jeremy Ma, senior manager of robotics at TRI, told The Robot Report. “Using an immersive telepresence system makes teaching and performing these tasks much more efficient, and we hope our progress can benefit others in the robotics community.”

Perception, navigation, and mobile manipulation in the household

Rather than map each home, as iRobot has done with recent floor-cleaning robots, TRI wanted its autonomous mobile robot to move around on its own as humans do.

The prototype robot chooses a path based on what it has learned, explained Dan Helmick, software engineer and senior robotics manager at TRI. He is also co-author of “A Mobile Manipulation System for One-Shot Teaching of Complex Tasks in Homes,” a study published by Cornell University. The robot can stop if it encounters something unexpected, Helmick said.

TRI’s mobile manipulator uses a combination of off-the-shelf and custom hardware. It includes four Intel RealSense cameras and two 5-megapixel Basler cameras for a wide field of view. The robot also has an ATI mini-45 force/torque sensor and runs on an 18-core Intel i9 CPU and an Nvidia TitanV graphics processing unit running Linux.

The vaguely humanoid robot uses industrial arms rather than collaborative arms, or cobots, because it is designed for ease of learning, not yet to operate alongside people, said TRI.

Joints in its Kinova Jaco2 arms and body provide 31 redundant degrees of freedom so that it can turn and reach items in, say, a kitchen cabinet, as needed. They include maxon motors and planetary gearheads. The robot also uses Sake Robotics grippers with under-actuated fingers, TRI said.

Toyota Research Institute teaches mobile manipulators

A major challenge for making robots smarter has been the massive amounts of data that they need to be trained on. TRI claimed that its methods, which include using simulations and virtual reality (VR) to train a robot to conduct certain tasks, require less data.

“More is not necessarily better,” said Helmick.

By teaching a robot about parameters rather than specific items, it can learn with just a few examples of what parts of an object are important for grasping, said Ma. The mobile manipulator can also learn how to move on uneven surfaces, interpret voice commands, and conduct complex tasks, he said.

After a single demonstration by a human using VR, the robot can autonomously perform multi-step tasks, said TRI. Toyota’s mobile manipulation system also includes dense visual representations of a scene with embedded objects, and a task graph of taught behaviors.

With simulations, a mobile manipulator can be trained to handle complex tasks such as loading a dishwasher, said Russ Tedrake, vice president of robotics research at TRI. Rather than train a robot on each item, it can learn how to reliably handle multiple objects.

TRI mobile manipulator software

TRI’s software architecture enables autonomous execution of taught tasks by processing visual and audio data, building a world model, mapping visual inputs to taught behaviors, and executing sequences of behaviors. Source: Toyota Research Institute

Sharing lessons with fleet learning

For robots to truly spread in people’s daily lives, they need to become useful for a wide variety of applications. Cloud computing, machine vision, and deep learning will lead to a flowering of capabilities similar to the “Cambrian Explosion” wrote TRI CEO Gill Pratt.

In short, the idea is that if one robot learns, that lesson can be shared among many robots rather than programming each one for a specific task or environment.

According to TRI, its mobile manipulator “can successfully perform a relatively complex human-level task about 85% of the time.” Each of the three tasks consisted of up to 45 behaviors, such as moving in a kitchen, opening and closing a refrigerator door, grasping a cup or container, and putting it on a table. Recoverable failures for individual behaviors — things the robot needed to do again — had a 99.6% success rate.


The Robot Report is launching the Healthcare Robotics Engineering Forum, which will be on Dec. 9-10 in Santa Clara, Calif. The conference and expo will focus on improving the design, development, and manufacture of next-generation healthcare robots. Learn more about the Healthcare Robotics Engineering Forum, and registration is now open.


Toyota Research Institute aims to help all service robots

Toyota Research Institute noted that its approach could be useful for teaching mobile manipulators in manufacturing, pick-and-place operations, packing, and logistics, but much work remains to be done for generalizing tasks. VR, teleoperation, and fleet learning could make robots more flexible and easier to instruct. TRI said it is improving key behaviors for all robots.

Rather than replacing human workers, TRI said it is working on robots to “amplify, assist, and empower human performance.” Toyota AI Ventures, which launched its second $100 million fund in May, has been investing in startups such as Freedom Robotics with similar goals.

In August, Toyota Motor Corp. announced a partnership with Preferred Networks Inc. to develop the Human Support Robot platform.

“We believe teaching a robot tasks is a promising first step to achieving our broader vision of fleet learning, specifically for assisting and empowering people in their home,” stated Toyota Research Institute.

About The Author

Eugene Demaitre

Eugene Demaitre is editorial director of the robotics group at WTWH Media. He was senior editor of The Robot Report from 2019 to 2020 and editorial director of Robotics 24/7 from 2020 to 2023. Prior to working at WTWH Media, Demaitre was an editor at BNA (now part of Bloomberg), Computerworld, TechTarget, and Robotics Business Review.

Demaitre has participated in robotics webcasts, podcasts, and conferences worldwide. He has a master's from the George Washington University and lives in the Boston area.

Comments

  1. Jérôme Bédard says

    October 7, 2019 at 3:20 pm

    Hello Eugene! Wonderful article!

    Just a Typo: it’s Kinova Jaco2 arm! (not a K).

    Thanks!

    Reply
    • Steve Crowe says

      October 7, 2019 at 3:27 pm

      Jerome, sorry about that. All set.

      Reply

Tell Us What You Think! Cancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Related Articles Read More >

limx robot in the foreground plcking an item from the ground.
TRON1 robot extends its reach with a new optional arm
Six of multiple possible assistance scenarios with a prototype of a new robot being developed at MIT. Top row: getting into/out of a bathtub, bending down to reach objects, and catching a fall. Bottom row: powered sit-to-stand transition from a toilet, lifting a person from the floor, and walking assistance.
MIT engineers create elder assist robot E-BAR to prevent falls at home
The Northeastern team that won the MassRobotics Form & Function Challenge.
Northeastern soft robotic arm wins MassRobotics Form & Function Challenge at Robotics Summit
A FANUC robot working in car manufacturing.
U.S. automotive industry increased robot installations by 10% in 2024

RBR50 Innovation Awards

“rr
EXPAND YOUR KNOWLEDGE AND STAY CONNECTED
Get the latest info on technologies, tools and strategies for Robotics Professionals.
The Robot Report Listing Database

Latest Episode of The Robot Report Podcast

Automated Warehouse Research Reports

Sponsored Content

  • Sager Electronics and its partners, logos shown here, will exhibit at the 2025 Robotics Summit & Expo. Sager Electronics to exhibit at the Robotics Summit & Expo
  • The Shift in Robotics: How Visual Perception is Separating Winners from the Pack
  • An AutoStore automated storage and retrieval grid. Webinar to provide automated storage and retrieval adoption advice
  • Smaller, tougher devices for evolving demands
  • Modular motors and gearboxes make product development simple
The Robot Report
  • Mobile Robot Guide
  • Collaborative Robotics Trends
  • Field Robotics Forum
  • Healthcare Robotics Engineering Forum
  • RoboBusiness Event
  • Robotics Summit & Expo
  • About The Robot Report
  • Subscribe
  • Contact Us

Copyright © 2025 WTWH Media LLC. All Rights Reserved. The material on this site may not be reproduced, distributed, transmitted, cached or otherwise used, except with the prior written permission of WTWH Media
Privacy Policy | Advertising | About Us

Search The Robot Report

  • Home
  • News
  • Technologies
    • Batteries / Power Supplies
    • Cameras / Imaging / Vision
    • Controllers
    • End Effectors
    • Microprocessors / SoCs
    • Motion Control
    • Sensors
    • Soft Robotics
    • Software / Simulation
  • Development
    • Artificial Intelligence
    • Human Robot Interaction / Haptics
    • Mobility / Navigation
    • Research
  • Robots
    • AGVs
    • AMRs
    • Consumer
    • Collaborative Robots
    • Drones
    • Humanoids
    • Industrial
    • Self-Driving Vehicles
    • Unmanned Maritime Systems
  • Business
    • Financial
      • Investments
      • Mergers & Acquisitions
      • Earnings
    • Markets
      • Agriculture
      • Healthcare
      • Logistics
      • Manufacturing
      • Mining
      • Security
    • RBR50
      • RBR50 Winners 2025
      • RBR50 Winners 2024
      • RBR50 Winners 2023
      • RBR50 Winners 2022
      • RBR50 Winners 2021
  • Resources
    • Automated Warehouse Research Reports
    • Digital Issues
    • eBooks
    • Publications
      • Automated Warehouse
      • Collaborative Robotics Trends
    • Search Robotics Database
    • Videos
    • Webinars / Digital Events
  • Events
    • RoboBusiness
    • Robotics Summit & Expo
    • DeviceTalks
    • R&D 100
    • Robotics Weeks
  • Podcast
    • Episodes
  • Advertise
  • Subscribe