Long-WAM: Scaling the Context of World-Action Models

TL;DR

Long-WAM is a model-system framework from NVIDIA + MIT + HKU + UCSD for scaling the visual context window of causal World-Action Models (WAMs) while keeping real-time control latency. The central finding: access to history is not the same as using it — longer histories pay off far more when the video foundation was pretrained autoregressively (AR), not bidirectionally. The recipe starts from LongLive-2.0’s AR checkpoint, continues autoregressive pretraining on ~10,000 hours of robot and egocentric video (RoVid-X, AgiBot World, EgoDex, EgoVerse, VITRA) without action labels, then adapts to WAM while preserving the history-to-future structure. On RoboCasa GR-1, pushing context from 0.0s to 19.2s raises success 63.3 → 78.7%, while a bidirectionally-pretrained initialization shows no net gain. Streaming encoding, asynchronous execution, and HW-specific acceleration give 107.4 ms per action chunk (including future-video latent prediction) on an RTX 5090, with real deployments on Unitree G1 and YAM — 95% success on dynamic cup stacking where π0.5 and Fast-WAM score 0/20.

Key claims

• Longer visual history helps only when the video foundation was pretrained autoregressively — on RoboCasa GR-1, context 0.0s → 19.2s raises success 63.3% → 78.7% for AR init; a bidirectional-pretraining init shows no net gain over the same sweep [§Abstract].
• Robot-domain AR continued pretraining on ~10k window-equivalent hours of robot + egocentric video (RoVid-X, AgiBot World, EgoDex, EgoVerse, VITRA), teacher-forced next-chunk prediction up to 30s with no action labels, further raises peak success on both RoboCasa GR-1 and LIBERO-Long [project page §Autoregressive video pretraining; §Abstract].
• Long-WAM reports the best results among compared methods on LIBERO-Long, RoboTwin 2.0, and DOMINO benchmarks [§Abstract].
• Real-time deployment is preserved by (a) streaming observation encoding, (b) asynchronous execution of future-video prediction vs. action emission, and (c) hardware-specific acceleration — enabling RTX 5090, DGX Spark, and Jetson AGX Thor targets without dropping future prediction [§Abstract].
• Per-action-chunk latency on RTX 5090 is 107.4 ms including future-video latent prediction [§Abstract].
• Real-robot deployment on Unitree G1 and YAM supports dynamic and long-horizon manipulation; headline datapoint: 95% success on dynamic cup stacking where π0.5 and Fast-WAM succeed in 0/20 trials each [§Abstract].
• Long-WAM composes with higher-level planning as a memory-informed executor on composite tasks rather than only operating as a standalone policy [§Abstract].

Method

Two-stage build. Stage 1 is autoregressive continued pretraining of the LongLive-2.0 video foundation on ~10,000 window-equivalent hours of robot + egocentric video (RoVid-X, AgiBot World, EgoDex, EgoVerse, VITRA), with teacher-forced next-chunk prediction on sequences up to 30 seconds and no action labels — the step that writes the history-to-future structure the WAM adaptation needs to preserve. Stage 2 adapts the AR video backbone to a World-Action Model while keeping that causal structure in the attention and prediction target, co-producing future-video latents and action chunks per step.

The system side is what makes long context compatible with real-time control: observations are encoded in a streaming fashion so the latency of a new frame doesn’t scale with history length; action-chunk emission runs asynchronously from the heavier future-video latent prediction so the control loop is not blocked by generation; and the inference stack is HW-tuned per target (RTX 5090, DGX Spark, Jetson AGX Thor). The reported RTX 5090 budget is 107.4 ms per action chunk including future-video latent prediction.

Results

• RoboCasa GR-1 (AR init): 63.3 → 78.7% as context sweeps 0.0 → 19.2 seconds; bidirectional init: ≈0 net gain over the same sweep [§Abstract].
• Peak success on RoboCasa GR-1 and LIBERO-Long raised further by adding robot-domain AR continued pretraining on top of the AR init [§Abstract].
• SOTA among compared methods on LIBERO-Long, RoboTwin 2.0, and DOMINO [§Abstract].
• RTX 5090 latency: 107.4 ms per action chunk (with future-video latent prediction) [§Abstract].
• Dynamic cup stacking on real robots: Long-WAM 95% vs π0.5 0% and Fast-WAM 0% (0/20 trials each) [§Abstract].

Why it’s interesting

Long-WAM is the first filed WAM to turn the generic “longer context helps” video-DiT finding into a conditional law for the robotics side: the win is only real if the video backbone was pretrained autoregressively. This directly engages the open question in Context Length / Quality Trade-off in Video Generation about whether the long-horizon operating points demonstrated by Context Forcing: Consistent Autoregressive Video Generation with Long Context, PackForcing: Short Video Training Suffices for Long Video Sampling and Long Context Inference, MALT Diffusion: Memory-Augmented Latent Transformers for Any-Length Video Generation, and LongLive: Real-time Interactive Long Video Generation transfer to policy learning, and gives a sharper answer than RoboTTT — 8k-timestep robot policy via test-time training (Jim Fan / NVIDIA GEAR)‘s RoboTTT and WAM-TTT: Steering World-Action Models by Watching Human Play at Test Time‘s weight-space history compression: it’s not just that long context is useful, it’s that pretraining-time causality determines whether a WAM can use it. The dynamic-cup-stacking 95% vs 0% result against both π0.5 and Fast-WAM: Do World Action Models Need Test-time Future Imagination? (which argued WAMs don’t need to imagine future video at inference) directly contradicts Fast-WAM’s “don’t bother imagining” conclusion when the task is reactive and long-horizon — reopening the question EgoWAM: World Action Models Beyond Pixels with In-the-Wild Egocentric Human Data already partially re-framed. Finally, as a real-time WAM with 107.4 ms per action chunk including future-video prediction, Long-WAM sits in the same latency-first regime as RealtimeWAM: One-Step Asynchronous World Action Models but attacks the budget via streaming encoding + async execution rather than one-step distillation.

See also

• Context Length / Quality Trade-off in Video Generation — Long-WAM is the first filed robotics datapoint where the AR-vs-bidirectional pretraining axis gates the long-context payoff.
• World Foundation Models — causal video foundation (LongLive-2.0) continued-pretrained on robot + egocentric data, then WAM-adapted.
• VLA Models — co-produces future-video latents and action chunks, deployed on Unitree G1 and YAM.
• Fast-WAM: Do World Action Models Need Test-time Future Imagination? — Fast-WAM; Long-WAM reports 0/20 for Fast-WAM on dynamic cup stacking, directly contesting Fast-WAM’s “no need to imagine future video” conclusion in reactive long-horizon tasks.
• EgoWAM: World Action Models Beyond Pixels with In-the-Wild Egocentric Human Data — closest controlled study on which world-model target transfers; Long-WAM’s AR-pretraining axis is complementary.
• RealtimeWAM: One-Step Asynchronous World Action Models — sibling real-time WAM attacking latency via one-step distillation + async expert dispatch rather than streaming encoding.
• LongLive: Real-time Interactive Long Video Generation — the LongLive video foundation Long-WAM builds on (via LongLive-2.0 → LongLive2.0-Robot).
• LongLive-2.0: An NVFP4 Parallel Infrastructure for Long Video Generation — LongLive-2.0 training infrastructure that enables the AR video base.
• RoboTTT — 8k-timestep robot policy via test-time training (Jim Fan / NVIDIA GEAR) — RoboTTT: alternative “history-into-weights” answer for robotics long context.
• WAM-TTT: Steering World-Action Models by Watching Human Play at Test Time — WAM-TTT: test-time steering via human-play weight updates.
• ACE-Ego-0: Unifying Egocentric Human and Robotic Data for VLA Pretraining, EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data — comparable egocentric + robot video pretraining corpora.