Zero-WAM: In-Context World-Action Modeling from Human Videos for Open-Ended Task Generalization

ListenShow, Don't Tell: Human Videos as Robot Prompts10:35

Show notes

Zero-WAM reframes zero-shot robotic generalization as a task-specification problem: a single causal video-action policy takes either a language instruction or an in-context human demonstration video, with no parameter update at test time. To feed it, the authors build HumanGen — 74.2K human-robot pairs across 8.6K tasks, synthesized by rewriting robot trajectories into human videos — and add an in-context future chunk prediction objective that forces the policy to actually watch the prompt. On seven held-out RoboTwin 2.0 tasks it reports 47.0% average success, a +29.5 point jump over the strongest video-action baseline. **Sources** - Zero-WAM: In-Context World-Action Modeling from Human Videos for Open-Ended Task Generalization

Subscribe to the channel: https://podcast.lumalabs.link/27a25f51ea81d41a8253f46c6133a35e/feed.xml

TL;DR

Zero-WAM (Robbyant / Ant Group; Zhou et al.) reframes zero-shot cross-task robotic generalization as an in-context task-specification problem: a single causal video-action policy accepts either a language instruction or an in-context human demonstration video as the task specification, and executes the task with no parameter update. To supply the paired data the recipe needs, the authors introduce an automatic pipeline that converts task-sampled robot trajectories into semantically matched human demonstration videos via VLM-driven view/prompt rewriting + image generation (Nano Banana, Qwen-Image) with a two-stage rating filter — yielding HumanGen, 74.2K human-robot ICL pairs across 8.6K tasks. Training adds an in-context future chunk prediction (IFP) objective that discourages the policy from shortcutting through robot history or text and forces it to draw task information from the human video prompt. On seven held-out RoboTwin 2.0 tasks Zero-WAM reports 47.0% average success — +29.5 absolute pp over the LingBot-VA baseline — and generalizes on real hardware to multi-object scenes, long-horizon manipulation, and fine-grained insertion.

Key claims

Zero-shot cross-task generalization is reframed as task specification: a single causal video-action policy is trained to accept either a language instruction or an in-context human demonstration video as the task specification, with no parameter update at test time [§Zero-WAM].
HumanGen contains 74.2K human-robot ICL pairs across 8.6K robotic tasks, generated by an automatic pipeline that samples tasks from robot corpora, rewrites the scene into a first-person human view via VLM prompting + Nano Banana / Qwen-Image generation, and filters through a two-stage 5/3/1 rating gate [§In-context Human Video Generation Pipeline].
The in-context future chunk prediction (IFP) objective is the load-bearing training-side lever — it suppresses shortcut learning from robot history and text and strengthens the policy’s use of the human video when predicting longer-term task evolution [§Zero-WAM].
On the RoboTwin 2.0 held-out suite of seven task-level unseen tasks (unseen object/container, bimanual, articulated, long-horizon), Zero-WAM reaches 46.95% average success, +29.50 pp over the LingBot-VA video-action baseline [§RoboTwin 2.0 Evaluation].
Real-hardware zero-shot testing covers task families that are hard to specify in language alone: precision-demand table-leg insertion (exact target configuration) and long-horizon sequential manipulation with a specified execution order (silver coffee cup → sponge → gourd) [§Zero-shot Testing].
Task-diverse video-action pretraining is constructed by repartitioning trajectories from five public datasets into >6,000 manipulation tasks and sampling at the task level rather than the trajectory level [§Task-diverse Data].

Method

Zero-WAM is a causal video-action policy that takes as input a task specification (either a language instruction or a variable-length human demonstration video) alongside the current robot observations, and generates action chunks. The core recipe has two halves. First, a data-generation pipeline: robot videos sampled from public corpora are captioned into task descriptions, a VLM rewrites the scene into a first-person human view preserving the manipulated objects and their spatial relationships, and image generators (Nano Banana, Qwen-Image) synthesize the initial human observation. A second stage generates the full human video and each generated pair is scored on a 5/3/1 scale — pairs are retained only if both the initial-frame rating and the video rating exceed threshold. This yields HumanGen: 74.2K human-robot ICL pairs spanning >8.6K tasks across public datasets, in-house platforms, simulation, and real-world collection.

Second, on the model side, standard action-chunk prediction is augmented with in-context future chunk prediction (IFP) — the policy is asked to predict longer-horizon future action chunks conditioned on the in-context prompt, which the authors argue suppresses shortcut learning from robot history and text and forces the policy to actually attend to the human demonstration. The training corpus mixes task-diverse video-action data (five public datasets repartitioned into >6,000 tasks and sampled at the task level) with the HumanGen ICL pairs. Downstream evaluation holds out seven RoboTwin 2.0 tasks at the task level (spanning unseen-object/container, bimanual, articulated, and long-horizon families), trains on the remaining 43, and measures zero-shot success. Real-hardware evaluation covers long-horizon sequential manipulation (with order specified by the human prompt) and precision-demand insertion (target configuration specified by the prompt).

Results

RoboTwin 2.0, task-level held-out (7/50): Zero-WAM reaches 46.95% average success, +29.50 pp over LingBot-VA (the strongest video-action baseline) [§RoboTwin 2.0 Evaluation].
HumanGen dataset scale: 74,200 human-robot ICL pairs across 8,600 tasks, sourced from >10 upstream datasets [§Zero-WAM headline figures].
Task-diverse video-action corpus: >6,000 manipulation tasks from five public datasets, sampled at the task level rather than trajectory level [§Task-diverse Data].
Real-hardware zero-shot: demonstrates cross-task generalization on multi-object long-horizon manipulation (silver coffee cup + sponge + gourd, order specified by human video) and precision-demand insertion (green + blue table legs into a white tabletop base) — regimes that language alone under-specifies [§Zero-shot Testing].

Why it’s interesting

Zero-WAM stakes out a sharpened version of the in-context-human-video position that GEN-1.5: Embodied Foundation Models are One-Shot Learners (GEN-1.5) claims as an emergent property of engine-scale pretraining — but Zero-WAM instead treats it as an explicit training problem, generating 74.2K aligned pairs via VLM-driven image synthesis and adding an IFP objective specifically to force attention onto the prompt video. This makes it the clearest filed counter to the “no explicit lever is needed at pretraining scale” reading of GEN-1.5 and to Behavior Prompting Policy: Demonstrations as Prompts for Manipulation BPP’s within-embodiment prompt conditioning, and it also directly contrasts with WAM-TTT: Steering World-Action Models by Watching Human Play at Test Time WAM-TTT (which absorbs the human video into a frozen model’s KV memory via TTT rather than into an in-context conditioning stream). The reported +29.5 pp margin over the same lab’s Causal World Modeling for Robot Control (LingBot-VA) LingBot-VA baseline is the sharpest single-lab head-to-head on file for the “human-video-as-in-context-prompt” recipe. The HumanGen generation pipeline is also a rare instance of synthetic human data — most syntheses in the Synthetic Training Data cluster generate robot data — extending the substrate the Human-to-Robot Retargeting recipes on this page compete over.