Cornell Develops AI System That Teaches Robots from Single Video Demonstrations

Researchers at Cornell University have introduced a new AI-based robotic learning framework named RHyME (Retrieval for Hybrid Imitation under Mismatched Execution). Unlike traditional robots that need detailed, repetitive training, RHyME enables robots to learn tasks from a single how-to video, even if the execution differs from what the robot can physically replicate. This breakthrough addresses a key limitation in robotic learning: the mismatch between human fluid motion and robotic capabilities.

RHyME allows robots to bridge these gaps by retrieving and referencing similar tasks stored in their memory, enabling adaptive learning with significantly less data. In practice, the system lets a robot interpret and perform complex multi-step actions, such as placing a mug in a sink, by drawing analogies from past learned behaviors. The researchers report that with just 30 minutes of training data, robots using RHyME achieved over 50% more successful task completions compared to older methods.

This innovation holds potential for faster and cheaper deployment of intelligent robotic systems in real-world environments. The paper describing this system, titled ‘One-Shot Imitation under Mismatched Execution,’ will be presented at a major robotics conference in May 2025.