1XWM: From Video to Action — 1X's video-pretrained world model as a NEO robot policy

TL;DR

1X repositions 1XWM from a policy evaluator (its earlier framing, see 1X World Model (1XWM) — action-controllable world model for humanoid policy evaluation) to a video-generation policy: at test time, a 14B text-conditioned video world model rolls out future frames from a starting image + text prompt, and a separate inverse dynamics model (IDM) extracts robot actions from the generated video for NEO to execute. The pitch is that internet-video pretraining plus a human-congruent hardware embodiment closes the “video knowledge → action” gap without large-scale teleoperated demonstrations, and generalizes to novel objects, two-handed coordination, and human interaction that were never in the robot training set. The post reports real-world success on both in-distribution and OOD tasks (30 trials each), that best-of-N over parallel video generations improves task success on ambiguous prompts, and that upsampled captions + egocentric human mid-training both improve video quality and real-world success on the hardest task tested (Scrub dish). Named limits: monocular pretraining leaves depth undershoot/overshoot even when generated videos “succeed”, and dextrous tasks (pouring, drawing) remain challenging.

Key claims

The system uses a two-stage grounding pipeline: a 14B generative video model rolls out future frames from (start frame, text prompt), then a separate IDM extracts an action trajectory for the robot to execute [§ “Raw video provides a visual what…”; § “At inference time…”].
The IDM is a two-image predictor with a sliding window of W=8 frames, using a Depth Anything backbone with a separate flow-matching head (not a single diffusion transformer as in DreamGen), trained on 400 h of unfiltered robot data including random play and motions that don’t correspond to any meaningful task [§ IDM description].
Backbone checkpoint selection uses a dynamic-time-warping distance between the IDM-decoded action sequence and ground-truth actions on generated videos — filtering for generations that both look good and yield accurate actions, not just look good visually [§ “To select the best 1XWM backbone checkpoint…”].
Post-training data mix is 98.5% high-quality tabletop pick-and-place with hands in view; two-handed coordination and human-interaction abilities emerge from the video-pretraining prior and egocentric-human mid-training rather than from robot data [§ “Our NEO post-training dataset…”; § “Next, we try tasks requiring two-handed coordination…”].
On both in-distribution and out-of-distribution tabletop tasks (30 trials each), 1XWM achieves stable success rates across diverse action primitives, though pouring and drawing remain challenging [§ “Beyond anecdotal examples…”].
Best-of-N video generation improves task success: on the “pull tissue” task, 1 → 8 parallel generations increases the best-of-8 selection success rate; executing “bad generations” hits ~0% success, so filtering (currently manual, amenable to VLM-based automation) is a load-bearing lever [§ “This makes us think that ideas like test-time compute…”].
Video-quality ablations on three 500-pair evaluation splits (In-distribution, New Tasks, T2I) show upsampled captions improve human-accept rate on every split; adding egocentric human data improves quality on New Tasks and T2I but is neutral or negative on In-distribution tasks where NEO already has good coverage [§ “We find that upsampling captions improves video generation quality…”; § “Adding egocentric human data improves generation quality…”].
Real-world ablations on the same two levers: on the hardest task (Scrub dish), the model with both egocentric mid-training and caption upsampling is the only one to achieve a nonzero success rate over 30 trials/model; on the in-distribution Grab-chips task all variants perform similarly [§ “On the only In-distribution task, Grab chips…”; § “Our Scrub dish task was the most challenging…”].
Backbone inference takes 11 s per 5-second video rollout (multi-GPU, built with Verda), with the IDM adding 1 s for action extraction — currently not real-time [§ “Building on the rapid progress…”].
Named failure: generated videos can be “overly optimistic about task completion and depth understanding” — visually plausible rollouts subtly violate object consistency, depth, geometry, or contact, so the real robot undershoots/overshoots even when the WM “succeeds”. Post-training reduces but does not eliminate this; monocular pretraining is named as the root cause and stereo/depth integration is flagged as future work [§ “Sometimes, generations tend to be overly optimistic…”].

Method

The runtime pipeline is: prompt + starting frame → 1XWM rolls out ~5 s of future video → IDM converts frames to a robot action trajectory (timewise-averaged across a batch of initial noise seeds and sliding windows for smoothness) → NEO executes the sequence. The 1XWM backbone is a 14B generative video model adapted to NEO’s embodiment via a multi-stage training strategy that combines caption upsampling (using a VLM to rewrite brief task descriptions into DALL-E-3-style descriptive captions), egocentric human mid-training, and NEO-specific post-training. The IDM diverges from DreamGen’s single-DiT design: two RGB frames at times t and t+8 are encoded through a Depth Anything backbone, and their embeddings condition a lightweight flow-matching head that predicts the action sequence — the depth encoder is what supplies geometric grounding to the extraction step. IDM training uses 400 h of unfiltered robot data (including random play), giving it broad action-tracking coverage rather than task-specific supervision.

At evaluation time, the team runs three types of measurement: (i) side-by-side WM-vs-REAL video alignment on individual tasks (qualitative), (ii) real-robot success rates on 30 trials/task for both in-distribution and OOD tasks, and (iii) 500-image-prompt human-annotator video-quality splits (In-distribution / New Tasks / T2I) that score physical plausibility, task completion, and embodiment consistency. Backbone checkpoint selection uses DTW distance between IDM-decoded action sequences on generated video and ground-truth actions, tying model selection back to the extraction stage rather than to visual similarity alone. Best-of-N over parallel video generations is validated as a test-time-compute lever specifically for prompts that occasionally produce off-task rollouts.

Results

Real-world 30-trial evaluations across in-distribution and OOD tabletop tasks show stable success across diverse action primitives; pouring and drawing remain challenging (no aggregate numbers on the blog).
Best-of-8 selection on “pull tissue” improves real success over 1-generation execution; executing generations judged as bad reaches ~0% success — filtering carries the gain.
Video-quality human-eval on 500-pair splits: upsampled captions improve accept rate on all three splits (In-distribution, New Tasks, T2I); egocentric-human mid-training improves accept rate on New Tasks and T2I, neutral-to-negative on In-distribution.
Real-world Scrub-dish ablation (30 trials/model): only the variant with both egocentric mid-training and caption upsampling achieves nonzero success; single-lever variants score zero. Grab-chips (in-distribution) shows no meaningful variation across ablation variants.
End-to-end latency: 11 s WM rollout (multi-GPU) + 1 s IDM per 5 s of generated video — not real-time; latency and closed-loop replanning with memory context are flagged as the next work items.

Why it’s interesting

This is the sharpest filed instance of the video-model-as-policy framing that the concept page tracks under World Foundation Models alongside Direct Video-Action Models — Causal Video Models Are Data-Efficient Robot Policy Learners (Rhoda’s DVA) and Causal World Modeling for Robot Control (LingBot-VA) (LingBot-VA). Unlike LingBot-VA’s unified causal AR flow-matching or DVA’s causal-video-as-policy, 1XWM keeps the video model text-conditioned and off-the-shelf-shaped (DreamGen/UniPi lineage) and puts the intelligence into two orthogonal levers — a Depth-Anything-conditioned IDM and DTW-based checkpoint selection — that could in principle wrap any bidirectional generative video backbone. The Scrub-dish ablation (both caption upsampling and egocentric mid-training required for nonzero success on the OOD task) is a rare non-trivial dependency between two data-side choices that most VLA papers report independently.

It also complements 1X’s earlier 1X World Model (1XWM) — action-controllable world model for humanoid policy evaluation framing (WM as policy evaluator): the same 1XWM brand now covers both the scoring and the acting roles from the World Foundation Models cluster’s deployment ledger — a single lab running both patterns in production is a useful datapoint against the framing that these are separate stacks. On the recipe-lever board of VLA Models, 1XWM stakes out a distinct point from action-pretraining-at-scale (π*0.6), clean-teleop (Spirit v1.5), unified-VLM pointing (Embodied-R1.5), and frozen-trace-WM + action expert (µ₀) — the “video model is the policy, IDM is a thin extractor” pattern. Alex’s Slack note also flags that 1X is now training on stereo vision for better spatial grounding — a direct response to the monocular-depth failure the blog itself names, and a concrete instance of the sensorimotor counter-position from Jitendra Malik: don't let CV researchers in robotics skip the sensorimotor level being absorbed into a video-first stack.