The Robot Report

  • Home
  • News
  • Technologies
    • Batteries / Power Supplies
    • Cameras / Imaging / Vision
    • Controllers
    • End Effectors
    • Microprocessors / SoCs
    • Motion Control
    • Sensors
    • Soft Robotics
    • Software / Simulation
  • Development
    • Artificial Intelligence
    • Human Robot Interaction / Haptics
    • Mobility / Navigation
    • Research
  • Robots
    • AGVs
    • AMRs
    • Consumer
    • Collaborative Robots
    • Drones
    • Humanoids
    • Industrial
    • Self-Driving Vehicles
    • Unmanned Maritime Systems
  • Business
    • Financial
      • Investments
      • Mergers & Acquisitions
      • Earnings
    • Markets
      • Agriculture
      • Healthcare
      • Logistics
      • Manufacturing
      • Mining
      • Security
    • RBR50
      • RBR50 Winners 2025
      • RBR50 Winners 2024
      • RBR50 Winners 2023
      • RBR50 Winners 2022
      • RBR50 Winners 2021
  • Resources
    • Automated Warehouse Research Reports
    • Digital Issues
    • eBooks
    • Publications
      • Automated Warehouse
      • Collaborative Robotics Trends
    • Search Robotics Database
    • Videos
    • Webinars / Digital Events
  • Events
    • RoboBusiness
    • Robotics Summit & Expo
    • DeviceTalks
    • R&D 100
    • Robotics Weeks
  • Podcast
    • Episodes
  • Advertise
  • Subscribe

Robotic prosthetics AI incorporates computer vision in NC State research

By The Robot Report Staff | May 30, 2020

Robotic prosthetics AI incorporates computer vision and uncertainty in NC State research

Imaging devices provide environmental context for robotic prosthetics. (a) On-glasses configuration using a Tobii Pro Glasses 2 eye tracker. (b) Lower limb data acquisition device. (c) and (d) Example frames from the cameras for the two configurations. (e) and (f) Example images of the experimental environment and terrains. Credit: Edgar Lobaton, NC State

Since bionic limbs often can’t rely on muscle contractions or nerve impulses to move as a normal arms or legs would, they need guidance from artificial intelligence. North Carolina State University researchers said they have developed software that can work with existing robotic prosthetics or exoskeletons to help people walk more naturally and safely over a variety of terrain.

The new software framework incorporates computer vision into prosthetic leg control, and it includes robust AI algorithms to better account for uncertainty.

“Lower-limb robotic prosthetics need to execute different behaviors based on the terrain users are walking on,” stated Edgar Lobaton, co-author of a paper on the work and an associate professor of electrical and computer engineering at North Carolina State University (NC State). “The framework we’ve created allows the AI in robotic prostheses to predict the type of terrain users will be stepping on, quantify the uncertainties associated with that prediction, and then incorporate that uncertainty into its decision-making.”

Developing an ‘environmental context’ for robotic prosthetics

The researchers focused on distinguishing between six different terrains that require adjustments in a robotic prosthetic’s behavior: tile, brick, concrete, grass, “upstairs,” and “downstairs.”

“If the degree of uncertainty is too high, the AI isn’t forced to make a questionable decision — it could instead notify the user that it doesn’t have enough confidence in its prediction to act, or it could default to a ‘safe’ mode,” said Boxuan Zhong, lead author of the paper and a recent Ph.D. graduate from NC State.

The new “environmental context” framework incorporates both hardware and software elements. The researchers designed the framework for use with any lower-limb robotic exoskeleton or robotic prosthetic device, but with one additional piece of hardware: a camera.

In their study, the researchers used cameras worn on eyeglasses and cameras mounted on the lower-limb prosthesis itself. The researchers evaluated how the AI was able to make use of computer vision data from both types of camera, separately and when used together.

“Incorporating computer vision into control software for wearable robotics is an exciting new area of research,” said Helen Huang, a co-author of the paper. “We found that using both cameras worked well, but required a great deal of computing power and may be cost prohibitive. However, we also found that using only the camera mounted on the lower limb worked pretty well – particularly for near-term predictions, such as what the terrain would be like for the next step or two.”

Huang is also the Jackson Family Distinguished Professor of Biomedical Engineering in the Joint Department of Biomedical Engineering at NC State and the University of North Carolina at Chapel Hill.

The 2020 Healthcare Robotics Engineering Forum is coming in September.


Model may benefit other deep learning applications

The most significant advance from the control model could be its relevance across AI.

“We came up with a better way to teach deep-learning systems how to evaluate and quantify uncertainty in a way that allows the system to incorporate uncertainty into its decision making,” Lobaton said. “This is certainly relevant for robotic prosthetics, but our work here could be applied to any type of deep-learning system.”

To train the AI system, researchers connected the cameras to able-bodied individuals, who then walked through a variety of indoor and outdoor environments. The researchers then did a proof-of-concept evaluation by having a person with lower-limb amputation wear the cameras while traversing the same environments.

“We found that the model can be appropriately transferred so the system can operate with subjects from different populations,” Lobaton says. “That means that the AI worked well even thought it was trained by one group of people and used by somebody different.”

Frameworks still needs testing on a robotic prosthetic

However, the new framework has not yet been tested in a robotic device. “We are excited to incorporate the framework into the control system for working robotic prosthetics – that’s the next step,” Huang said.

“And we’re also planning to work on ways to make the system more efficient, in terms of requiring less visual data input and less data processing,” said Zhong.

The paper, “Environmental Context Prediction for Lower Limb Prostheses with Uncertainty Quantification,” was published in IEEE Transactions on Automation Science and Engineering. The paper was co-authored by Rafael da Silva, a Ph.D. student at NC State;, and Minhan Li, a Ph.D. student in the Joint Department of Biomedical Engineering. The work was done with support from grants from the National Science Foundation.

Tell Us What You Think! Cancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Related Articles Read More >

Artedrone's Sasha autonomous mechanical thrombectomy system.
Microrobot system is designed to float inside stroke patient for autonomous thrombectomy
A man walking down a crosswalk wearing the Ekso personal exoskeleton with a woman walking beside him. The man is also using crutches to stay steady.
NVIDIA accepts Ekso Bionics into its Connect program
RealMan Robotics offers a variety of mobile manipulators.
RealMan displays embodied robotics at Automate 2025
Six of multiple possible assistance scenarios with a prototype of a new robot being developed at MIT. Top row: getting into/out of a bathtub, bending down to reach objects, and catching a fall. Bottom row: powered sit-to-stand transition from a toilet, lifting a person from the floor, and walking assistance.
MIT engineers create elder assist robot E-BAR to prevent falls at home

RBR50 Innovation Awards

“rr
EXPAND YOUR KNOWLEDGE AND STAY CONNECTED
Get the latest info on technologies, tools and strategies for Robotics Professionals.
The Robot Report Listing Database

Latest Episode of The Robot Report Podcast

Automated Warehouse Research Reports

Sponsored Content

  • Sager Electronics and its partners, logos shown here, will exhibit at the 2025 Robotics Summit & Expo. Sager Electronics to exhibit at the Robotics Summit & Expo
  • The Shift in Robotics: How Visual Perception is Separating Winners from the Pack
  • An AutoStore automated storage and retrieval grid. Webinar to provide automated storage and retrieval adoption advice
  • Smaller, tougher devices for evolving demands
  • Modular motors and gearboxes make product development simple
The Robot Report
  • Mobile Robot Guide
  • Collaborative Robotics Trends
  • Field Robotics Forum
  • Healthcare Robotics Engineering Forum
  • RoboBusiness Event
  • Robotics Summit & Expo
  • About The Robot Report
  • Subscribe
  • Contact Us

Copyright © 2025 WTWH Media LLC. All Rights Reserved. The material on this site may not be reproduced, distributed, transmitted, cached or otherwise used, except with the prior written permission of WTWH Media
Privacy Policy | Advertising | About Us

Search The Robot Report

  • Home
  • News
  • Technologies
    • Batteries / Power Supplies
    • Cameras / Imaging / Vision
    • Controllers
    • End Effectors
    • Microprocessors / SoCs
    • Motion Control
    • Sensors
    • Soft Robotics
    • Software / Simulation
  • Development
    • Artificial Intelligence
    • Human Robot Interaction / Haptics
    • Mobility / Navigation
    • Research
  • Robots
    • AGVs
    • AMRs
    • Consumer
    • Collaborative Robots
    • Drones
    • Humanoids
    • Industrial
    • Self-Driving Vehicles
    • Unmanned Maritime Systems
  • Business
    • Financial
      • Investments
      • Mergers & Acquisitions
      • Earnings
    • Markets
      • Agriculture
      • Healthcare
      • Logistics
      • Manufacturing
      • Mining
      • Security
    • RBR50
      • RBR50 Winners 2025
      • RBR50 Winners 2024
      • RBR50 Winners 2023
      • RBR50 Winners 2022
      • RBR50 Winners 2021
  • Resources
    • Automated Warehouse Research Reports
    • Digital Issues
    • eBooks
    • Publications
      • Automated Warehouse
      • Collaborative Robotics Trends
    • Search Robotics Database
    • Videos
    • Webinars / Digital Events
  • Events
    • RoboBusiness
    • Robotics Summit & Expo
    • DeviceTalks
    • R&D 100
    • Robotics Weeks
  • Podcast
    • Episodes
  • Advertise
  • Subscribe