Amazon launched SageMaker Reinforcement Learning (RL) Kubeflow Components, an open-source toolkit designed to help manage robotics workflows. SageMaker RL is designed to make it faster to develop machine learning capabilities for everything from perception to controls and optimization.

Amazon SageMaker is a service built on Amazon’s two decades of experience developing real-world machine learning applications. SageMaker RL builds onto SageMaker, adding pre-packaged RL toolkits and and integration with simulation environments included in AWS RoboMaker. SageMaker supports various machine learning frameworks, including TensorFlow, PyTorch and mxnet.

“Robots are being used more widely in society for purposes that are increasing in sophistication, such as complex assembly, picking and packing, last-mile delivery, environmental monitoring, search and rescue, and assisted surgery,” Amazon wrote in a blog introducing SageMaker RL. “Robotics often involves training complex sequences of behaviors. RL is an emerging ML technique that can help develop solutions for exactly these kinds of problems. It learns complex behaviors without requiring any labeled training data, and can make short-term decisions while optimizing for a long-term goal.

“For example, when a robot interacts with its environment, this mostly takes place in a simulator. The robot receives a positive or negative reward for actions that it takes. Rewards are computed by a user-defined function that outputs a numeric representation of the actions that should be incentivized. The agent tries to maximize positive rewards, and as a result the model learns an optimal strategy for decision-making.”

To highlight the capabilities of SageMaker RL, Amazon shared a case study about Woodside Energy, a natural gas producer in Australia. It is using machine learning methods for robotic manipulation. Woodside built a robot, Ripley, that performs a “double block and bleed, a manual pump shutdown procedure that involves turning multiple valves in sequence.” Ripley consists of two UR5 collaborative robot arms from Universal Robots, a Clearpath Robotics Husky mobile robot base, Intel RealSense D435 cameras on each wrist, and a Kodak PixPro body camera.

The RL formulation uses the joint states and camera views as inputs to the agent and outputs optimal trajectories for valve manipulation. You can watch a demo of Ripley performing a double block and bleed in the video below.

“Our team and our partners wanted to start exploring using machine learning methods for robotics manipulation,” says Kyle Saltmarsh, Robotics Engineer at Woodside Energy. “Before we could do this effectively, we needed a framework that would allow us to train, test, tune, and deploy these models efficiently. Utilizing Kubeflow components and pipelines with SageMaker and RoboMaker provides us with this framework and we are excited to have our roboticists and data scientists focus their efforts and time on algorithms and implementation.”