Listen to this article

To celebrate Boston Dynamics joining the Hyundai Motor Group family, Spot joined BTS for a dance off. We documented how that dance developed from an idea to reality—the goals, the inspirations, and the challenges of choreographing robots. Kathleen Brandes, a software engineer on the Spot team, notes, “Every time we make a new video, the ability of the robot is improved. It looks fluid and artistic on film, but we need to make it work on hardware first.”

What is the technical path from artistic vision to hardware? The engineers and programmers working on the project used a few different techniques that employed both Spot’s Choreographer software and its API to produce the variety of behaviors in the finished dance.

Setting the steps in Choreographer

Choreographer, a tool for authoring and controlling movement through Spot’s API, became available last year for customers in media and entertainment. Choreographer is designed to bridge the gap between the creative process of a traditional choreographer and the technical process of programming a robot.

Working with human dancers, a choreographer can give relatively high-level instruction—perform this step three times, for example—rather than dictating the specific joint angle or torque required. The dancer translates these instructions into detailed motions of the body. But the details are exactly the language required by traditional robotics. Similarly, if the choreographer asks a dancer to move in a way that’s not physically possible, the dancer can use their innate understanding of their own body to modify the steps based on their capabilities and environment. By contrast, a typical robot will adhere strictly to the directive—even if it falls.

The Choreographer software allows non-technical users to create dances like a choreographer instructing a dancer rather than like an engineer programming a robot. The Choreographer controller understands Spot’s physics and environment, prioritizing balance first and then following the specified steps. With this athletic intelligence baked into the robot, you can simply provide higher-level instructions while Spot does the rest. If asked to do something physically impossible, or if faced with an environmental challenge like a slippery floor, Spot will find the possible motion most similar to what was requested and do that instead—analogously to what a human dancer would do.

We took inspiration from common video and audio editing software to develop an interface that enables you to specify the desired robot motion quickly, intuitively, and at your preferred level of detail. A dance is created as a series of configurable moves visualized on a timeline. These “move blocks” represent holistically optimized requests to the controller, rather than pre-scripted motion. In addition to enabling Spot to prioritize balance over strict adherence to a defined motion, this approach also allows you to combine multiple moves simultaneously. The timeline is divided into separate tracks for the legs, body, arm, and gripper, so you can independently specify a move for each track.

A directive to “take three steps” is as simple as placing three “step” blocks on the timeline.

To make the middle step slower, you can extend the step so it takes more width on the timeline.

If a little more detail is desired, we provide that through parameters. A directive like “step a little higher” can be given by dragging a slider.

Combining moves to do something like “step while swaying” is accomplished by placing moves on multiple tracks at the same time in the timeline.

Finally, you set the beats per minute, automatically syncing the timeline to whatever music will be used and keeping Spot on beat for the final performance. Check out our documentation for more detailed information on the Choreographer software.

Expanding with animation

For those situations where you have an exact output in mind and don’t mind getting a little more technical, we also developed an animation API, which will be available in our upcoming 2.4 release. This pipeline allows you to create wholly custom sequences using 3D animation tools and integrate them in Choreographer scripts just like the default moves. For the intro sequence of the “Spot’s On It” video, we used Autodesk Maya to produce the kaleidoscoping dance moves. Autodesk Maya is a 3D animation software that gives fine-grain control for authoring and editing kinematics trajectories, but a range of toolsets can be used with this API.

Just as the software enables a choreographer to focus on the steps, rather than their mechanics, the animation API enables an animator to stay focused on the overall look and feel of a sequence. Robotic movement consultant Jakob Welner explains, “In traditional animation you are free to design the exact movement that you want the audience to see, but when handling robots, there are many layers of computation and physical constraints to consider. However, the animation API allows you to run plausible animations easily, without worrying about the connotations and constraints. Spot will do what is possible and automatically try to correct the rest.” This high level of control is possible because the animation is still treated as a request to the balancing controller, freeing you from having to produce animations with perfect physics.

Bringing it all together

These different approaches to authoring dance sequences can be combined and layered in Choreographer to create the intricate results showcased in the “Spot’s On It” video. For example, in the “Ripples” sequence with seven Spots performing a wave-like dance, we used both animation and standard Choreographer workflows. We set the steps and body positions with stock move blocks, but wanted a more fluid effect in the arm motions. Animation allowed us to get precisely the desired arm movement, while the layering framework enabled us to combine the animated arm motion with the stock moves, resulting in the best of both worlds.

With these different design techniques in conjunction with Spot’s built-in athletic intelligence, it’s easy to create dances that look fluid and natural, without overtaxing either your choreographer or your robot.

About the Author

Eric Whitman is a senior roboticist at Boston Dynamics, where he works on controls and perception for the Spot robot. Before coming to Boston Dynamics, he received a Ph.D. in robotics from Carnegie Mellon University, where his research focused on bipedal robotic walking.