Masked Visual Actions for Unified World Modeling

TL;DR

Masked Visual Actions (MVA) proposes a single pixel-space control interface for video-based world models: an action is expressed as a partially revealed trajectory of an arbitrary entity in a video, and the model fills in the rest. Revealing the robot’s own motion turns the model into a forward-dynamics predictor (scene response to low-level robot actions); revealing desired object motion turns the same weights into an inverse model that synthesizes the robot behavior needed to achieve that outcome. A single checkpoint, finetuned on only ~15 hours of masked examples from real videos and simulation, is reported to serve three downstream uses — imagined rollouts for policy evaluation, ranking candidate futures for model-based planning, and inverse modeling from desired object motion [Abstract].

Key claims

The central design decision is how to inject action into a video model: MVA argues action should live in the same visual space where the video model already learned its interaction priors — as visibility over a partial trajectory of an entity in the frame — rather than as an external text/vector conditioner [Abstract].
The same interface expresses forward and inverse control by choice of which entity’s trajectory is revealed: robot-motion reveal → forward dynamics; object-motion reveal → recovered robot behavior [Abstract].
Training data footprint is small: only ~15 hours of masked examples drawn from real videos and simulation are used to finetune a video backbone into the MVA regime [Abstract].
A single checkpoint reportedly achieves strong visual fidelity and controllability across diverse scenes and multiple embodiments — no per-embodiment or per-scene head is needed [Abstract].
Downstream use 1 — policy evaluation: the model’s imagined rollouts have outcomes that correlate with real-world execution, positioning MVA as a video-WFM-based policy-evaluation substrate [Abstract].
Downstream use 2 — model-based planning: MVA ranks candidate futures well enough to improve downstream decision-making [Abstract].
Downstream use 3 — inverse modeling: MVA synthesizes robot motion consistent with a desired object motion supplied as a partial reveal [Abstract].

Method

MVA takes a pretrained video model and finetunes it under a masking objective: given a video clip, mask out most of an entity’s trajectory (either the robot’s or an object’s) and require the model to complete the video conditioned on the visible partial trajectory. Because the trajectory is drawn/rendered directly in pixel space, no auxiliary tokenizer, camera calibration, or action-space alignment is needed — the “action” is a shape in the same image the video model already reads.

At inference, the same checkpoint is used in three modes by choosing what to reveal:

Forward mode (reveal robot motion): the model predicts the scene’s response frame-by-frame, acting as a forward dynamics model for low-level robot actions.
Inverse mode (reveal object motion): the model completes the video with a plausible robot motion that produces that object trajectory — an inverse-dynamics / behavior-synthesis pass.
Rank-many-rollouts mode (planning): multiple candidate action reveals are generated, the model rolls each out, and downstream criteria pick a winner.

The abstract does not specify the video backbone, the exact masking scheme, or the ~15h source split between real and simulation — the body is the natural place to look for those details, which were not accessible at filing time from the abstract page alone.

Results

The abstract reports:

Fidelity/controllability: strong across diverse scenes and multiple embodiments from a single checkpoint.
Policy evaluation: imagined-rollout outcomes correlate with real-world execution (no r-value or benchmark named at the abstract level).
Planning: candidate-future ranking improves decision-making (no benchmark or SR delta named at the abstract level).
Inverse modeling: robot motion is synthesized from desired object motion (qualitative claim at abstract level).

Numerical baselines, benchmark names, and comparison partners are not disclosed in the abstract — the body would be needed for like-for-like comparisons against DriftWorld: Fast World Modeling through Drifting, RoboWorld: Fast and Reliable Neural Simulators for Generalist Robot Policy Evaluation, μ₀: A Scalable 3D Interaction-Trace World Model, and ImageWAM: Do World Action Models Really Need Video Generation, or Just Image Editing?.

Why it’s interesting

MVA adds a new answer to the WFM cluster’s central design question — how do we communicate action to a video model whose priors were learned action-free? — and it does so by refusing to leave the pixel space at all. That places it in explicit contrast with three lines the wiki already tracks:

µ₀ (μ₀: A Scalable 3D Interaction-Trace World Model) uses 3D interaction traces (B-spline control points over salient contact/object/hand points) as the action interface, with a frozen world model paired to an embodiment-specific action expert. MVA collapses the same “trajectory as action” idea into 2D pixel visibility, so the interface is even cheaper and requires no 3D lifting — but presumably at the cost of the 3D affordances µ₀ gets for free.
ImageWAM (ImageWAM: Do World Action Models Really Need Video Generation, or Just Image Editing?) argues the world model should be an image editor whose KV cache (not decoded pixels) feeds an action expert. MVA takes the opposite stance: the pixels are the interface, and the same generative rollout is what serves forward, inverse, and planning modes. MVA is a “one-model, three-uses” bet vs ImageWAM’s “editor as prior + separate action head”.
DriftWorld (DriftWorld: Fast World Modeling through Drifting, shared author Yilun Du) is a speed answer to fast policy evaluation and MPC-style search over candidate action sequences; MVA is an interface answer to what an “action” is in the first place. The two are compatible — a drifting one-step generator with an MVA-style pixel-trajectory conditioner is a natural next step.

MVA is also a partial answer to Sitzmann’s action-conditional-WFM chicken-and-egg (The flavor of the bitter lesson for computer vision): the interface itself supplies training signal from unpaired video (any entity trajectory in any clip can be masked and used as an MVA supervision pair), reducing the need for paired action–observation data. And within the wiki’s WFM deployment-pattern ledger, MVA introduces a ninth pattern (after backbone, predictive-surprise reward, interruptible tool, policy-internal substrate, edit-prior via KV cache, frozen trace-WM + small action expert, value-model, inference-time guidance signal): WFM-as-pixel-trajectory-completer, where forward, inverse, and planning are unified as different mask conditionings of one generative video pass.