Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrations

TL;DR

RIGVid (Robots Imitating Generated Videos) does zero-shot robotic manipulation by generating a single task-specific video from an off-the-shelf video diffusion model and physically replaying it on a real robot — no demonstrations, no robot-specific training, no policy learning. The pipeline is: (1) prompt a video generator with the initial RGB-D scene + a language command, (2) filter unsuccessful generations with a VLM, (3) recover per-frame depth, segment the manipulated object, and track its 6D pose across the generated video with FoundationPose, then (4) retarget that 6D object trajectory to the robot. Filtered generated videos prove as effective as real human demonstrations, and downstream performance scales with the generator’s video quality — making this one of the cleanest existence proofs that frontier video models are already usable as action-supervision sources for embodied manipulation.

Key claims

Generated videos, after VLM filtering, are as effective as human demonstration videos as the visual-imitation supervision signal — robots can act entirely from synthetic supervision [Abstract].
RIGVid is embodiment-agnostic: the same recovered 6D object trajectory drives xArm7, ALOHA, and a bimanual setup; ALOHA achieves 80% success on pouring vs 100% on the default xArm setup [§Extensions, App.].
The pipeline outperforms more compact alternatives — keypoint prediction by VLMs and dense feature-point tracking are both dominated by full 6D pose tracking from the generated video [Abstract].
Performance improves monotonically with generator quality, suggesting that capability gains in video diffusion models translate directly into manipulation gains without re-engineering the rest of the stack [Abstract, §Results].
A VLM filter is reliable enough to gate generated videos before execution: failure modes (the generated video does not actually carry out the command) can be rejected with high accuracy [§Method, Fig. 1].
The system handles real-time disturbances during execution by re-tracking the object pose at deployment time, decoupling the open-loop trajectory plan from the closed-loop replay [§Method].

Method

RIGVid takes an RGB-D observation of the scene plus a free-form language command (e.g. “pour water on the plant”). A state-of-the-art video diffusion model generates one or more candidate videos depicting the task being performed in the captured scene. A VLM is then queried per candidate to determine whether the video actually follows the command; failed generations are discarded. The surviving video is passed through a monocular depth estimator to recover per-frame depth, the manipulated object is segmented, and FoundationPose tracks its 6D pose across all frames to produce a 6D object pose trajectory. After grasping the object, the trajectory is retargeted to the robot’s end effector in an embodiment-agnostic way. At deployment, the robot tracks the actual object pose in real time and adjusts its motion to stay on the planned trajectory, handling disturbances and execution-time variations.

The framing is deliberately minimal: the only learned components in the pipeline are off-the-shelf (the video model, the VLM, the depth estimator, FoundationPose). There is no policy training and no robot-specific data collection. This is what lets the authors claim the system inherits, rather than amortizes, the underlying generators’ capabilities.

Results

Real-world evaluation covers four manipulation tasks — pouring water, lifting a lid, placing a spatula on a pan, and sweeping trash — chosen to span depth variation (minimal in pouring, large in lifting), thin/partially occluded objects (spatula, sweeping brush), and diverse geometries. Headline numbers: filtered generated videos match human demonstration videos as the supervision signal; performance tracks generator quality; xArm7 hits 100% pouring success, ALOHA 80% on the same task without retargeting changes [§Extensions]. The system generalizes to a bimanual setup that simultaneously places a pair of shoes. The full pipeline beats two more compact baselines explicitly: VLM keypoint prediction (no video generation) and dense feature-point tracking (no 6D pose) [Abstract].

Exact per-task success-rate tables and the filtering-rate quantification are in the arxiv version (2507.00990); the project page leads with qualitative robot rollouts and ablations on tracking modality.

Why it’s interesting

This is the cleanest filed instance of using a video diffusion model directly as a world model in the LeCun sense — a generator whose rollout is rich enough to drive embodied action without first being distilled into a policy. It complements World Foundation Models from the opposite direction: where Inference-time Physics Alignment of Video Generative Models with Latent World Models uses VJEPA-2’s predictive surprise (latent-side) as a frozen reward to align video generators, RIGVid uses the generated pixels themselves (rollout-side) as the action prior, and asks only that a 6D tracker can read motion out of them. The two together pin down the empirical disagreement Sitzmann’s bitter-lesson essay names (The flavor of the bitter lesson for computer vision) — RIGVid is the strongest filed data point that the generative-rollout side of WFMs is already usable as a substrate for embodied tasks, no explicit 3D state required. It also contrasts with the VLA-pipeline pattern in NVIDIA Unveils New Open Models, Data and Tools to Advance AI Across Every Industry (Cosmos Reason 2 as reasoning backbone for Isaac GR00T N1.6): RIGVid skips the policy head entirely and reads actions out of the synthesized video, raising the question of when a learned VLA is even needed.