Robots now handle glass, reflective items with simple cameras

Researchers at Tokyo University of Science have developed HEAPGrasp, a vision-based robotic system that enables accurate grasping of transparent and reflective objects using only a standard RGB camera, without relying on depth sensors. This innovation addresses a longstanding challenge in automated material handling, where objects like glass, shiny metals, and clear plastics have been difficult for robots to detect and manipulate due to distortions and confusion in conventional 3D sensing methods. HEAPGrasp reconstructs 3D object shapes by capturing visual outlines or silhouettes from multiple angles and applying a Shape from Silhouette technique, which remains effective despite transparency or glare. The system employs semantic segmentation via deep learning to isolate objects from the background and uses a planning algorithm to optimize camera movement, reducing unnecessary repositioning while maintaining accuracy. Tested across 20 scenarios involving various transparent, opaque, and reflective items, HEAPGrasp achieved a 96% success rate with a single camera, cutting camera movement by 52% and execution time by 19%