Zhaoning Wang, Siyi Chen, Jun Gao
University of Michigan
Each scene is one recorded robot episode. Left of the line is the real robot; right of it, its physics twin in Isaac Sim replays the same recorded actions. Point at a scene to move the line, click it for the full videos, and scroll sideways for more.
From one recorded demonstration to a physics scene that replays it. The robot arm itself aligns the cameras, and an agent runs each tool and checks the result before moving on. The steps play on their own; drag the scene to look around.
GPT-6 judge: accepted the real video left of the line, the replay right of it
The verified twin also gives a robot agent its context. From the replayed demonstration and the scene's measurements we write a demonstration context: the person's sequence as tool calls, and measured numbers relative to the objects. The agent (ORA) then runs the task twice on new layouts, once with the context and once without it; its model weights never change.
A person teaches the task once on the SO-101 arm. 2× speed.
Its replay in simulation reproduces the demonstration. 2× speed.
Measured in the twin: what the agent reads, not new weights.
The agent reads the context and runs the task on a new real layout: both blocks in. Pauses cut, 4× speed.
The same agent on a new real layout, without the context: no block goes in. Pauses cut, 4× speed.
Success on 10 unseen layouts in the twin
| Put the blocks in the containerSuccess: at the end, the twin's own check finds both blocks inside the container. | Pull out the drawerSuccess: the drawer ends at least 5.5 cm out, its travel measured in the twin. | |
|---|---|---|
| ORA (baseline) | 8/10 | 7/10 |
| + demonstration context | 10/10 | 9/10 |
The same task on the real SO-101 arm with a new layout, once with the context and once without; blue and orange blocks stand in for the green and red ones. One run each, so these are demonstrations, not success rates. Pauses are cut and the videos run at 4× speed.
Our twins cover rigid objects and simple articulated objects, such as drawers. Soft bodies, such as towels and napkins, and complex articulated objects are left for future work.