RealtimeWAM: One-Step Asynchronous World Action Models

TL;DR

RealtimeWAM (NTU + SenseTime + collaborators) attacks the two latency bottlenecks of imagine-then-execute World Action Models — intra-expert iteration (multi-step action denoising) and inter-expert waiting (sequential video-then-action execution in the shared Mixture-of-Transformers backbone) — with two orthogonal techniques stacked on an existing WAM. Teacher-Anchored Consistency Distillation (TACD) supplements the standard local consistency objective with explicit supervision from a frozen teacher’s multi-step rollout endpoint, closing the local–global error gap that otherwise makes naive one-step action distillation drift. Cross-Expert Wavefront Pipelining (CEWP) overlaps the video and action experts by sharing the video KV cache block-by-block, synchronizing only immediately before the corresponding action attention consumes it. Reported ~25× end-to-end speedup on H100 at <1% drop across LIBERO, LIBERO-Plus, and RoboTwin, applied to both Fast-WAM and Faster-WAM variants.

Key claims

• The two latency bottlenecks of MoT-based WAMs are structurally distinct: intra-expert iteration (action expert runs ~10 denoising steps) and inter-expert waiting (action expert is idle while video expert generates future frames) [Abstract].
• Low local consistency error alone does not guarantee accurate final actions in one-step distilled WAMs; TACD closes this local–global error gap by anchoring the student against the frozen teacher’s multi-step rollout endpoint [Abstract].
• CEWP overlaps the two experts via block-wise sharing of the video KV cache, synchronizing only immediately before action attention consumes each block — the per-block coupling replaces whole-frame waiting [Abstract].
• Combined recipe delivers ~25× end-to-end speedup on an H100 at <1% drop across LIBERO, LIBERO-Plus, and RoboTwin benchmarks [Abstract].
• The technique composes across WAM variants: both Fast-WAM and Faster-WAM derivatives demonstrate the superiority of RealtimeWAM [Abstract].

Method

RealtimeWAM operates on an existing Mixture-of-Transformers WAM in the Fast-WAM / Faster-WAM lineage, where a video DiT and an action expert share attention across the backbone. Baseline inference is sequential: the video expert runs its denoising loop first, then the action expert runs its own multi-step loop conditioned on the video expert’s output (or shared representations). RealtimeWAM collapses both axes.

Along the intra-expert axis, Teacher-Anchored Consistency Distillation trains a single-step action student against a frozen multi-step teacher. Standard consistency distillation enforces that denoising from any point on the ODE trajectory returns the same endpoint, but this is a local objective — small local errors can compound into bad final actions when run for only one step. TACD adds an explicit anchor: the student’s one-step prediction is supervised against the frozen teacher’s full multi-step rollout endpoint, decoupling single-step quality from accumulated-error drift.

Along the inter-expert axis, Cross-Expert Wavefront Pipelining splits the video expert’s forward pass into blocks and streams the resulting KV cache to the action expert block-by-block rather than waiting for the whole video rollout to complete. Synchronization is deferred to the moment the action expert’s attention module actually needs the next block, producing a wavefront-style overlap between the two experts’ forward passes.

Benchmarks: LIBERO and LIBERO-Plus (simulation manipulation suites), RoboTwin (bimanual simulation benchmark), evaluated across Fast-WAM and Faster-WAM variants.

Results

• End-to-end speedup: ~25× on an H100 (specific baseline denoising-step count and NFE budget not in the abstract excerpt) [Abstract].
• Task-success drop: under 1% across LIBERO, LIBERO-Plus, and RoboTwin [Abstract].
• Applied to both Fast-WAM and Faster-WAM, demonstrating recipe transferability within the WAM family [Abstract].

(Per-benchmark splits, per-component ablation tables, and head-to-head latency numbers against C³ache / DiagDistill / πR² / REMAC on comparable hardware are not reproduced in the abstract excerpt; a /bud refresh after fetching the full PDF would surface the ablation and the exact baseline NFE/denoising-step counts the 25× is measured against.)

Why it’s interesting

RealtimeWAM is the first filed WAM recipe that attacks both intra-expert and inter-expert latency axes simultaneously in a shared Mixture-of-Transformers backbone — a structural answer to the question Fast-WAM: Do World Action Models Need Test-time Future Imagination? left open after reporting a 190 ms Fast-WAM and 810 ms Fast-WAM-IDM: once the WAM’s architectural future-imagination choice is fixed, how far can distillation plus pipelining push the resulting latency? The ~25× figure, if measured against a baseline comparable to Fast-WAM-IDM’s 810 ms, puts WAM inference solidly into the sub-50 ms regime that FLUX 3 Action: A 7B World Action Model for Robot Control FLUX 3 Action and APXinf-robo — Edge Inference Engine for VLA/WAM Models on Jetson Thor / Orin APXinf-robo have been staking out for VLA-class policies via step distillation + FP8 and quantized serving respectively.

The TACD contribution is a specific answer to the problem that Transition Matching Distillation for Fast Video Generation TMD identified for few-step diffusion students — standard local consistency objectives silently diverge from inference-time multi-step behavior — now applied to the action branch of a WAM rather than to video. Where TMD closes the train/inference gap by rolling out the student’s flow head during DMD gradient computation, TACD closes a different formulation of the same gap by anchoring against the frozen teacher’s full multi-step endpoint during consistency training. The CEWP contribution extends the async-inference line pioneered by πR²: Reactive Real-time Flow Policies πR² and Real-Time Robot Execution with Masked Action Chunking REMAC from between policy chunks to between the two experts of a single forward pass — a finer-grained parallelism axis than any filed async-VLA recipe so far. The recipe-lever-board placement is clean: distillation (TACD) sits in the diffusion-distillation column, pipelining (CEWP) sits in the async-inference column of VLA-serving, and the paper’s bet is that the two axes compose multiplicatively on top of a shared MoT WAM backbone.

See also

• Fast-WAM: Do World Action Models Need Test-time Future Imagination? — Fast-WAM is the baseline MoT WAM architecture RealtimeWAM is explicitly built on; same two-expert MoT substrate on a video-DiT backbone, same imagine-then-execute framing
• FLUX 3 Action: A 7B World Action Model for Robot Control — FLUX 3 Action’s guidance+step distillation stack is the sibling industrial-scale WAM latency recipe (4→1 step on 7B joint video+action backbone, 32.29 ms per prediction in FP8); RealtimeWAM is the matched research-scale answer with explicit CEWP pipelining
• C³ache: Accelerating World Action Models with Cross Inference Chunk Cache — C³ache is the training-free cross-chunk-residual-caching WAM speedup counterpart to RealtimeWAM’s within-chunk recipe; the two attack orthogonal temporal axes and should compose
• Transition Matching Distillation for Fast Video Generation — TMD closes the DMD train/inference gap for video via student flow-head rollout; TACD closes the analogous gap for one-step action distillation via frozen-teacher endpoint anchoring
• πR²: Reactive Real-time Flow Policies — πR² reorganizes flow-matching VLA inference across policy chunks via asynchronous conditioning + latency-adaptive staircase; CEWP applies the same async-pipelining principle within a single WAM forward pass across its two experts
• Real-Time Robot Execution with Masked Action Chunking — REMAC is the training-time recipe for async-inference robustness in flow-matching VLAs; RealtimeWAM extends the async axis one layer finer into the inter-expert pipeline
• World Foundation Models — adds a new WAM-serving row to the recipe-lever board — the first filed instance of combined intra-expert step distillation plus inter-expert pipelining
• VLA Models — RealtimeWAM is the WAM-side counterpart to the VLA-serving optimizations tracked here, pushing MoT WAM inference into VLA-class latency regimes
• Diffusion Distillation — TACD is a new single-step action-distillation objective addressing the local–global error gap specifically for WAM action experts