Robust mobile manipulation, which is starting to appear in supply chain operations, is a step toward service robots with more generalized capabilities. Toyota Research Institute last week posted a video (above) showing the state of its research on household robots.

Aging populations in Europe, East Asia, and North America pose challenges, according to a United Nations study cited by Toyota Research Institute (TRI). One of the goals of service robotics is to enable people to “age in place,” or remain in their homes with as much autonomy as possible and without scarce human aid.

Unlike other technology enterprises and automakers working on self-driving cars, Toyota has positioned itself as a “mobility company” as it looks to diversify its offerings beyond vehicles and into assisting households. As part of that effort, the company set an internal challenge of teaching robots to perform a variety of tasks in actual homes.

“Teaching robots to perform tasks is a key element of advancing fleet learning for enabling useful robot assistants in homes,” Jeremy Ma, senior manager of robotics at TRI, told The Robot Report. “Using an immersive telepresence system makes teaching and performing these tasks much more efficient, and we hope our progress can benefit others in the robotics community.”

Perception, navigation, and mobile manipulation in the household

Rather than map each home, as iRobot has done with recent floor-cleaning robots, TRI wanted its autonomous mobile robot to move around on its own as humans do.

The prototype robot chooses a path based on what it has learned, explained Dan Helmick, software engineer and senior robotics manager at TRI. He is also co-author of “A Mobile Manipulation System for One-Shot Teaching of Complex Tasks in Homes,” a study published by Cornell University. The robot can stop if it encounters something unexpected, Helmick said.

TRI’s mobile manipulator uses a combination of off-the-shelf and custom hardware. It includes four Intel RealSense cameras and two 5-megapixel Basler cameras for a wide field of view. The robot also has an ATI mini-45 force/torque sensor and runs on an 18-core Intel i9 CPU and an Nvidia TitanV graphics processing unit running Linux.

The vaguely humanoid robot uses industrial arms rather than collaborative arms, or cobots, because it is designed for ease of learning, not yet to operate alongside people, said TRI.

Joints in its Kinova Jaco2 arms and body provide 31 redundant degrees of freedom so that it can turn and reach items in, say, a kitchen cabinet, as needed. They include maxon motors and planetary gearheads. The robot also uses Sake Robotics grippers with under-actuated fingers, TRI said.

Toyota Research Institute teaches mobile manipulators

A major challenge for making robots smarter has been the massive amounts of data that they need to be trained on. TRI claimed that its methods, which include using simulations and virtual reality (VR) to train a robot to conduct certain tasks, require less data.

“More is not necessarily better,” said Helmick.

By teaching a robot about parameters rather than specific items, it can learn with just a few examples of what parts of an object are important for grasping, said Ma. The mobile manipulator can also learn how to move on uneven surfaces, interpret voice commands, and conduct complex tasks, he said.

After a single demonstration by a human using VR, the robot can autonomously perform multi-step tasks, said TRI. Toyota’s mobile manipulation system also includes dense visual representations of a scene with embedded objects, and a task graph of taught behaviors.

With simulations, a mobile manipulator can be trained to handle complex tasks such as loading a dishwasher, said Russ Tedrake, vice president of robotics research at TRI. Rather than train a robot on each item, it can learn how to reliably handle multiple objects.

Sharing lessons with fleet learning

For robots to truly spread in people’s daily lives, they need to become useful for a wide variety of applications. Cloud computing, machine vision, and deep learning will lead to a flowering of capabilities similar to the “Cambrian Explosion” wrote TRI CEO Gill Pratt.

In short, the idea is that if one robot learns, that lesson can be shared among many robots rather than programming each one for a specific task or environment.

According to TRI, its mobile manipulator “can successfully perform a relatively complex human-level task about 85% of the time.” Each of the three tasks consisted of up to 45 behaviors, such as moving in a kitchen, opening and closing a refrigerator door, grasping a cup or container, and putting it on a table. Recoverable failures for individual behaviors — things the robot needed to do again — had a 99.6% success rate.

---

The Robot Report is launching the Healthcare Robotics Engineering Forum, which will be on Dec. 9-10 in Santa Clara, Calif. The conference and expo will focus on improving the design, development, and manufacture of next-generation healthcare robots. Learn more about the Healthcare Robotics Engineering Forum, and registration is now open.

---

Toyota Research Institute aims to help all service robots

Toyota Research Institute noted that its approach could be useful for teaching mobile manipulators in manufacturing, pick-and-place operations, packing, and logistics, but much work remains to be done for generalizing tasks. VR, teleoperation, and fleet learning could make robots more flexible and easier to instruct. TRI said it is improving key behaviors for all robots.

Rather than replacing human workers, TRI said it is working on robots to “amplify, assist, and empower human performance.” Toyota AI Ventures, which launched its second $100 million fund in May, has been investing in startups such as Freedom Robotics with similar goals.

In August, Toyota Motor Corp. announced a partnership with Preferred Networks Inc. to develop the Human Support Robot platform.

“We believe teaching a robot tasks is a promising first step to achieving our broader vision of fleet learning, specifically for assisting and empowering people in their home,” stated Toyota Research Institute.