Dream.exe: Can Video Generation Models Dream Executable Robot Manipulation?

TL;DR

Dream.exe is an evaluation framework that probes whether video generation models have internalized physical laws by executing the motion they depict on a robot in simulation. Given a scene image and a task description, the pipeline synthesizes a manipulation video, converts the generated motion into a robot trajectory, runs it in a physics simulator, and reports execution success alongside visual quality and trajectory fidelity. The benchmark covers 101 manually curated manipulation tasks at three levels of physical complexity and scores 8 models spanning frontier closed-source generators, open-source generators, and robot-specific models. The headline finding is that several models achieve measurable execution success — generative priors do encode some real physical knowledge — but visual quality is a poor predictor of executability, exposing a capability axis that standard visual evaluations miss.

Key claims

Visual quality is a poor predictor of executability: across the 8 evaluated video models, the ranking on visual metrics diverges from the ranking on execution success, so passing standard visual evals is not sufficient to be a usable manipulation-video generator [Abstract].
Several models achieve measurable execution success on the 101-task benchmark, indicating that internet-scale video pretraining already encodes physical knowledge meaningful enough to drive a robot in simulation, not just to render plausible pixels [Abstract].
The framework operationalizes “world model” capability via a concrete video-to-execution pipeline — generate video → extract motion → produce trajectory → execute in physics simulator → measure task success — rather than via human or VLM judgments of physical plausibility [Abstract].
The benchmark spans 101 manually curated tasks at three levels of physical complexity and scores models on three orthogonal axes (visual quality, trajectory fidelity, execution success), with the latter providing the grounding signal absent from purely visual metrics [Abstract].
The 8-model panel mixes frontier closed-source generators, open-source generators, and robot-specific models, enabling direct comparison between general-purpose video models and models that were specifically trained on robot data [Abstract].

Method

Dream.exe is a video-to-execution pipeline. The input is a scene image and a natural-language task description. A video generation model under test produces a manipulation video showing the task being performed in the captured scene. The system extracts the motion depicted in that video, converts it into a robot trajectory, and executes the trajectory in a physics simulator. The simulator returns whether the task succeeded — i.e. whether the dreamed motion, when literally replayed by a simulated robot, produces the intended end state in a physics engine. This success signal is paired with two purely visual axes (visual quality of the generated video, and trajectory fidelity between the generated motion and a reference) so that each model is characterized by a three-vector rather than a single visual score.

The benchmark is 101 manually curated manipulation tasks organized at three levels of physical complexity, evaluated across 8 models — frontier closed-source generators, open-source generators, and robot-specific generators. Code is announced as planned open-source release at the showlab/Dream.exe repository (not yet released at filing time).

Results

The abstract reports two headline findings. First, several models in the 8-model panel achieve measurable execution success on the 101 manipulation tasks — strong evidence that internet-scale video pretraining already encodes physical priors with non-trivial actionable content, not just visually plausible motion. Second, the per-model ranking on visual quality does not predict the ranking on execution success, so judging a video generator by FVD, VBench-style visual metrics, or human-preference Elo alone systematically misses a capability axis that the embodied evaluation surfaces. The abstract does not break out the per-model rank table or the per-complexity-level success rates; those are in the full paper.

Why it’s interesting

Dream.exe is the closest filed methodological companion to Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrations (RIGVid): both run the same loop — generate video → recover motion → execute on robot — but with opposite framings. RIGVid uses one frontier generator to drive a real robot end-to-end as a policy-free manipulation system; Dream.exe runs the same pipeline across 8 generators in simulation to use execution success as a benchmark signal. The pair is mutually reinforcing: RIGVid is an existence proof that the loop works at all on real hardware, Dream.exe is the systematic measurement showing the loop separates frontier video models in a way visual quality does not. Together they argue that the WFM-as-rollout reading defended in The flavor of the bitter lesson for computer vision is not just plausible but already differentially measurable across model families.

This also slots cleanly into the embodied-evaluation axis of Video Generation Benchmarks alongside Rethinking Video Generation Model for the Embodied World (ReVidGen / RBench / RoVid-X) (RBench), which uses automatic VLM-style scoring of robot-oriented video output. Dream.exe is the harder, more grounded variant of RBench’s question: where RBench asks “do reviewers think this looks like a successful robot manipulation,” Dream.exe asks “does a simulator confirm it.” Comparing the two leaderboards on overlapping models would directly test whether VLM-judge benchmarks for robot video can reach simulator-grounded execution success, an open question called out on the Video Generation Benchmarks page. It also contrasts with Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation (PhyGenBench) (PhyGenBench), which probes physical commonsense at the per-clip level via VLM judges of plausibility rather than via downstream task execution.