Self-Improving Video-to-Action Models

TL;DR

Mirror (Sitzmann group, MIT) is a self-supervised recipe for learning the video-to-action / inverse-dynamics mapping that lets a video world model act as a robot policy — without any expert action labels, hand-crafted rewards, or demonstration-action pairs. An agent queries a frozen video planner for a video of how the task would be completed, runs its current (initially random) V2A model to try to execute that plan in the environment, and the resulting (observed-video, executed-action) pairs train the next iteration of the V2A model. Across Atari, OGBench, DeepMind Control, and Minecraft BASALT, Mirror beats learning-from-observation baselines with learned video plans and matches near-expert performance with oracle plans; in Minecraft survival it is reported as the first method to acquire an iron pickaxe through self-improvement alone, with no rewards or expert actions.

Key claims

Self-supervised training of a V2A / inverse-dynamics model emerges from the loop “query video planner → execute best-guess actions → train V2A on the resulting (video, action) pairs” without any expert action labels or hand-crafted rewards [§ Abstract].
The video planner stays frozen throughout; only the V2A model is updated across iterations [§ 1 Introduction, § Abstract].
The approach is characterized as neither reinforcement learning nor standard behavior cloning — it is a form of behavior cloning that does not require video-action pairs because the required pairs are produced by the agent’s own interactions with the environment [tweet thread; § 1 Introduction].
Sample efficiency comes from the planner re-focusing the agent on whatever the current V2A model does not yet execute correctly: small stochasticity in the V2A does not hurt in general because the planner re-plans every chunk, but on the subset where it fails, the planner surfaces exactly the failure cases needed as new training data [tweet thread].
On Atari, OGBench, and DeepMind Control, Mirror outperforms leading learning-from-observation methods when using learned video plans, and approaches near-expert performance when using oracle video plans [§ Abstract, § Experiments].
On Minecraft BASALT, Mirror outperforms methods with access to expert actions or hand-engineered task knowledge [§ Abstract].
In Minecraft survival, Mirror is the first method to acquire an iron pickaxe through self-improvement alone, without rewards or expert action labels [§ Abstract, § Minecraft Survival].
In the offline setting (expert action labels available but no environment interaction), the same recipe sets a new state of the art on obtaining diamonds in Minecraft [§ Abstract, § Offline Minecraft].
An emergent in-context property of the video planner is reported: dropped in a lava lake, Mirror reaches 76% cumulative escape rate over 30 attempts with history retained vs. 46% when context is reset after each try — the video planner biases future sampling toward strategies that previously succeeded without any fine-tuning [tweet thread, @ericmchen1 sibling thread].

Method

The two moving pieces are a frozen pre-trained video planner and a trainable video-to-action (V2A) model, sometimes called an inverse dynamics model. The agent executes an outer loop over the environment: given the current state, it queries the video planner for a short video clip of how the task would be completed, runs the current V2A model on that clip to produce an action sequence, and executes those actions in the environment. The resulting pair — the actually observed video and the actions that produced it — is a valid training sample for the V2A model, regardless of whether those actions matched what the planner had in mind. The V2A model is then retrained on the growing dataset of such pairs. Early iterations produce near-random actions because the V2A is near-random, but the recorded (video, action) pairs are still internally consistent and the V2A gets slightly better at reproducing videos; subsequent iterations improve monotonically. Because the planner is frozen and keeps steering the agent toward unsolved behavior, the exploration pressure is placed into the planner’s output distribution rather than into a reward signal. The authors emphasize this as neither RL (no reward) nor conventional behavior cloning (no action-labeled demonstrations). An ambitious engineering requirement is that the video planner be both strong (SOTA on the relevant distribution) and fast enough to run inside a real-time control loop; the paper reports the code runs on both NVIDIA and AMD GPUs (MI350X), supported by AMD’s university research program.

Results

Navigation (environments unspecified in abstract-level detail): ~30 minutes of real-time gameplay suffice for the agent to learn to follow navigation plans [tweet thread].
Minecraft iron pickaxe: on the order of 5 hours of gameplay to collect sufficient action labels to craft the iron pickaxe through self-improvement alone, without rewards or expert actions — reported as a first-of-its-kind result [§ Minecraft Survival, tweet thread].
Atari / OGBench / DeepMind Control: Mirror beats leading learning-from-observation baselines when using learned video plans; with oracle video plans the policy approaches near-expert performance [§ Abstract].
Minecraft BASALT: outperforms methods given expert actions or hand-engineered task knowledge [§ Abstract].
Offline Minecraft (expert actions, no interaction): new state of the art on obtaining diamonds [§ Abstract].
Lava-lake escape probe (emergent in-context behavior of the video planner): 76% cumulative escape rate over 30 attempts with context retained vs. 46% with context reset after each attempt — ×1.65 lift from in-context learning in the planner alone, no fine-tuning [tweet thread, @ericmchen1 (8/10)].

Why it’s interesting

Mirror is the sharpest filed instance of using a video world model as the policy, with a trained V2A as the thin translator (World Foundation Models, VLA Models) that also solves the data-labeling bottleneck end-to-end: both Rhoda’s DVA (Direct Video-Action Models — Causal Video Models Are Data-Efficient Robot Policy Learners) and VERA (video model → faithful IDM, cited by Sitzmann on the same site) still require robot-action data to train the IDM; Mirror removes that requirement by generating the (video, action) pairs through the control loop itself. It also plants a flag on the “neither RL nor BC” territory: the loop structurally resembles the data→model→data recipes in Self-Play Data Generation (Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing InfTool, Playful Agentic Robot Learning RATs, ASPIRE: Agentic /Skills Discovery for Robotics ASPIRE) — but the verifier is the video planner’s own distribution rather than an environment reward or a judge model, which is a structurally new verification source for this cluster. The emergent in-context lava-escape behavior is the second filed instance (after RATs’ persistent skill library) of a self-improvement loop where the frozen generator-side component carries non-trivial state across attempts. The sibling Vincent Sitzmann piece The flavor of the bitter lesson for computer vision is the normative argument this paper operationalizes: video pre-training as the embodied-AI substrate, with no explicit 3D state, and the V2A as the small embodiment-specific layer. If the “~5 hours of gameplay to iron pickaxe” number holds up outside Minecraft, this is a concrete counter to the demonstration-scaling orthodoxy that currently dominates the VLA Models recipe-lever board.