World Pilot: Steering Vision-Language-Action Models with World-Action Priors

TL;DR

World Pilot augments a Vision-Language-Action policy with two complementary priors drawn from a separate World-Action Model (WAM): a Latent Steering path that conditions the perception layer on a scene-evolution latent, and an Action Steering path that supplies an anticipated trajectory as a motion prior to the action generator. The framing argues VLM-pretrained VLAs inherit static image-text semantics but miss the continuous, contact-rich dynamics of manipulation, and that routing dynamics in through two channels — perception-side and action-side — recovers most of that gap without retraining the VLM. Reports state-of-the-art 84.7% Total success rate on the LIBERO-Plus zero-shot OOD benchmark and the highest success rate on every real-robot setting across four manipulation tasks, with the largest margins under shifts in viewpoint, geometry, deformable state, and pose. Notably the scene-evolution prior remains effective even when supplied by a video-pretrained world model that has not been action-post-trained, so the dynamics signal does not require paired action data.

Key claims

Two complementary steering pathways (Latent Steering on perception + Action Steering on the action generator) outperform either alone, with the WAM providing both an anticipated view of the scene and a trajectory-level motion hint alongside the VLA’s semantic conditioning [Abstract].
World Pilot attains 84.7% Total success on LIBERO-Plus zero-shot OOD, the SOTA at submission [Abstract].
Largest absolute margins vs. baselines occur under distribution shifts in viewpoint, geometry, deformable state, and pose — categories where the VLM’s static-pretraining priors are weakest [Abstract].
The scene-evolution latent transfers from a video-pretrained world model that was not action-post-trained, indicating the perception-side prior does not depend on paired action–observation data [Abstract].
Real-robot evaluation across four manipulation tasks reports the highest success rate on every setting tested [Abstract].

Method

World Pilot is a VLA framework augmented with priors from a separately-trained World-Action Model. Two routing pathways feed the WAM’s outputs into the policy’s decision chain:

Latent Steering: a scene-evolution latent from the WAM is injected into the VLA’s perception layer, giving the policy an anticipated view of how the scene will evolve before the action head fires. Critically, this latent transfers from a purely video-pretrained world model — no action-conditioned fine-tuning is required.
Action Steering: an anticipated trajectory from the WAM is supplied as a motion prior to the action generator, biasing the per-step action toward the WAM’s predicted continuation.

The two paths are complementary: Latent Steering conditions what the policy sees and Action Steering conditions what the policy is about to do. The VLM backbone itself is not modified. Full method content beyond the abstract was not retrievable at filing time (ar5iv HTML body returned only the metadata block); a /bud refresh once the full PDF is parsed would sharpen the architectural details and ablation breakdown.

Results

LIBERO-Plus zero-shot OOD: 84.7% Total success, reported as SOTA at submission [Abstract].
Real-robot, 4 manipulation tasks: highest success rate on every setting evaluated [Abstract].
Distribution-shift breakdown: largest margins vs. baselines on viewpoint, geometry, deformable-state, and pose shifts [Abstract].
Ablation framing: scene-evolution prior remains effective when the supplying WAM is video-pretrained-only (no action post-training), suggesting the perception-side gain is not gated on paired action–observation data [Abstract].

Quantitative tables and per-shift numbers were not retrievable at filing time.

Why it’s interesting

World Pilot is the cleanest recently-filed instance of the WFM-as-policy-peer recipe, sitting beside two other patterns the wiki tracks: μ₀: A Scalable 3D Interaction-Trace World Model (µ₀’s frozen 3D-trace world model + small action expert) and ImageWAM: Do World Action Models Really Need Video Generation, or Just Image Editing? (ImageWAM’s pretrained image editor consumed via KV cache). All three argue the VLA should not be a monolithic VLM-with-action-head; they differ in what the WAM outputs (3D traces vs. image edits vs. anticipated frame+trajectory) and where it plugs in (action expert input vs. KV cache vs. dual perception+action steering). World Pilot is the only one of the three that injects on both sides simultaneously. It also gives a concrete answer to the chicken-and-egg of action-conditional WFMs flagged by Sitzmann (The flavor of the bitter lesson for computer vision) and re-stated as an open question on World Foundation Models — by showing the perception-side prior transfers from a video-only WAM, World Pilot decouples the WFM training data from the policy’s action data. The viewpoint/geometry/deformable/pose breakdown also lines up with the OOD failure modes Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models and π*0.6: a VLA That Learns From Experience (RECAP) separately target with different levers, making this paper a useful third axis on the VLA-generalization debate.