Between workforce challenges and accelerated demand for e-commerce order fulfillment as well as social distancing during the COVID-19 pandemic, the need for automated picking and sorting of thousands of SKUs is greater than ever. However, traditional industrial robots are not flexible or easy enough to deploy, according to Intel Corp. One example of a solution using computer vision with robotics hardware and software is RightPick2 from RightHand Robotics Inc.

With e-commerce growing by 15% to 20% annually and faster during the pandemic, warehouses face increasing demand for single-item picking, according to Intel. Some facilities have turned to semi-automated totes, batch picking, and pick walls, but such systems may not reduce human touch points or be able to do complex bin picking, added Somerville, Mass.-based RightHand Robotics.

“An e-commerce fulfillment center may have 100,000 to 1 million different products, and it’s really hard to use traditional robotics there,” stated Vince Martinelli, head of product and marketing at RightHand. “Industrial robots typically have a very structured environment and a very limited number of products that they’re going to try to handle. In an e-commerce environment, items come jumbled in a bin with a mix of products — they’re changing all the time. This requires a new capability.”

RightPick2 uses RealSense D415 for computer vision

Computer vision has evolved to help robots with piece-picking, singulation, and packing challenges, said Joel Hagberg, head of product management and marketing at Intel RealSense.

“A traditional robotics solution struggles to handle more than a few distinct objects and is usually limited in the number, shape or type of objects it can recognize and pick,” he told The Robot Report. “While machine learning can help train a system to pick individual items reliably, what’s needed is a way to pick any unknown item, without training.”

RightHand Robotics said it designed RightPick2 to automate more stages of e-commerce fulfillment for grocery, pharmacy, retail, and more. Intel said its RealSense depth cameras provide data enabling the robots to pick individual items from mixed bins and pick them with damaging them using collision avoidance.

“We call this the hand/eye coordination side of our system,” said Martinelli. “We use the Intel RealSense D415 depth camera as our primary vision system. It’s vital for segmentation and all aspects of motion planning.”

Robust data key to reliable picking

RightHand gathered data from millions of picks to learn the best ways to approach different shapes and classes of items and the optimal ways to orient them for efficient sorting and lifting. Intel claimed that the RealSense D415, part of its D400 series, has a field of view that offers a higher depth resolution for small objects or situations in which precise measurements are needed, such as in bin picking.

“The Intel RealSense Depth Camera D415 used in the RightPick2 is a stereo depth camera with an integrated RGB camera,” explained Hagberg. “The camera has a z error — also known as depth error — of less than 2% at 2m or less. The depth pixel size is 1.4μm × 1.4μm. This combination of high resolution with low error helps to generate an accurate depth image for any customer application.”

“Stereo depth cameras lose accuracy over time due to loss of calibration between the two imagers caused by shock or vibration,” he said. “With the introduction of our new self-calibration feature in the SDK [software developers kit], developers can test and recalibrate the sensors in the field as quickly as 0.6 sec. This feature can also run automatically, ensuring a more reliable data stream for our customers.”

“The Intel RealSense Depth camera D415 includes an integrated D4 Vision Processor,” Hagberg said. “This vision processor performs all depth calculations directly on the device. This processor is optimized specifically for depth calculations, making it extremely fast. This results in a low-power solution ideal for any autonomous robot without requiring additional processing power.”

Intel noted that the D415’s compact design and price-performance ratio allow RightHand Robotics to use multiple cameras to collect robust data.

“Using multiple cameras helps to generate accurate object understanding by viewing items from many angles,” said Hagberg. “More accurate object understanding results in reliable picking in a variety of situations.”

Intel offers developers ease of integration

“Intel RealSense offers a variety of options to make integration easy,” said Hagberg. “For developers wishing to rapidly prototype and test robots, the self-contained plug-and-play depth cameras can be directly mounted on a robotic prototype.”

“For those looking to maximize efficiency, we also offer modules which can be built into higher-volume products, offering the best price and performance,” he added. “One easy-to-use SDK across the entire line of devices makes it possible for developers to focus on their own solution.”

“The SDK is open-source and has cross-platform support for Windows, Android, and Linux, as well as other popular platforms like Raspberry Pi, and [it] supports development using ROS, C/C++, Python, and more,” Hagberg said. “This gives developers the flexibility to work how they want to with Intel RealSense cameras, as well as get started quickly with a broad library of code samples, how-to articles, and useful software like the Intel RealSense Viewer and debug tools.”

“The advantage of a well-supported open-source platform is the complete access developers have to a constantly improving codebase as well as the flexibility to modify it to their own needs,” he said. “The growing library of code samples will help get any project up and running fast with some of the most crucial applications for robotics developers like collision avoidance, occupancy mapping, and path planning.”

“By using the same SDK across the entire Intel RealSense portfolio, any application developed for one camera will work for any future camera with minimal changes to the application code. Develop once, and take advantage of a wide range of existing and future devices without the need to fork or develop different branches for different devices.”

RightHand said its RightPick2 is optimal for kitting, in which separate items are packaged as one unit, as well as for sorter induction and goods-to-picker tending. The robotic systems can sort batch-picked items, those coming out of automated storage and retrieval systems (ASRS), and facilitate order quality assurance, it said.

“The key to making the shift to automated piece picking is computer vision.” said Martinelli.