The DAWN of World-Action Interactive Models

TL;DR

DAWN (COWARobot + SJTU + Hohai) is a latent generative model for end-to-end autonomous driving that recursively couples world prediction and action denoising at inference time. The paper formalizes World-Action Interactive Models (WAIMs) — a class of World Action Models where future world states and future actions co-evolve as coupled variables rather than being inferred in parallel or in a fixed predict-then-plan pipeline. DAWN instantiates this with a V-JEPA 2 Large visual backbone, an Auto-Encoder Resampler that compresses dense tokens into compact latent world tokens, a causal-Transformer World Predictor, and a DiT-style World-Conditioned Action Denoiser that share weights between an initial action-proposal role and an interactive refinement role. On NAVSIM v1 it sets a new perception-free PDMS of 89.1 (best NC, EP, TTC among perception-free methods); on nuScenes it reduces the average L2 trajectory error to 0.33 m vs WorldRFT’s 0.47 m, with the best average collision rate (0.11%).

Key claims

Existing WAMs structurally decouple world generation from action generation — either parallel heads from shared visual context or a one-way predict-then-plan pipeline — and therefore cannot model the action-contingent nature of decision-relevant futures in interactive driving [§1, Fig. 1].
A WAIM is defined as a WAM that infers future world W* and future action A* as a self-consistent coupled pair via iterative interaction W^(t+1) = p(W | obs, A^t), A^(t+1) = p(A | obs, W^(t+1)), rather than marginalizing over independently-generated futures [§2.1, Eq. 4–5].
DAWN’s inference performs a short latent world rollout (not pixel-space rendering), with the World Predictor and World-Conditioned Action Denoiser sharing weights between a “proposal” role (initial action from encoder context) and a “refinement” role (action conditioned on predictor rollout); the only difference is the role-specific query embeddings and the input source [§2.2, Eq. 9].
On NAVSIM v1, perception-free DAWN reaches PDMS 89.1 (best perception-free, also best NC=98.7, EP=84.3, TTC=96.0 in that class) vs Drive-JEPA at 89.0 and the perception-free LAW/World4Drive baselines at 83.8/85.1 [Table 1].
On nuScenes end-to-end planning, DAWN obtains the lowest L2 error at all horizons (1s 0.17, 2s 0.31, 3s 0.52, Avg 0.33 m) vs the next-best WorldRFT 0.47 m average, with the best average collision rate 0.11% [Table 2].
Component ablation isolates the WAIM gain: adding the Resampler alone (compact latent) does not improve PDMS (82.9→82.8); adding the World Predictor (explicit future rollout) lifts to 85.2; adding interactive world-action coupling lifts to 87.9 [Table 8].
Disabling either direction of coupling weakens the model: removing World→Action drops PDMS 87.9→81.6, removing Action→World drops 87.9→84.9 — supporting the WAIM principle that world evolution and action generation must mutually constrain each other [Table 5].
Interactive refinement saturates around 4 rounds and most of the rollout-horizon gain (PDMS 82.8 zero-rollout → 87.9 at 4s) is already captured at 2–3s latent horizon, indicating the world branch acts as a short action-relevant dynamics hypothesis rather than a full future simulator [Fig. 3, Table 6].

Method

DAWN operates in a compact semantic latent space. The Student Vision-Encoder (V-JEPA 2 Large) extracts dense visual tokens from the current camera observation, which an Auto-Encoder Resampler compresses to a compact latent world representation z_W. A training-time Teacher Vision-Encoder + resampler produces target future latents that supervise the world branch and is dropped at inference.

The core is the recursive interaction of two modules:

World Predictor — a causal Transformer that predicts future latent world tokens from the current latent context and the current action hypothesis.
World-Conditioned Action Denoiser — a DiT-style diffusion module that denoises action tokens conditioned on both the latent context and the predicted future world. It uses 5 sampling steps at inference. It plays two roles with shared weights: an initial proposal (queried by q_prop, conditioned on encoder context only) and an interactive refinement (queried by q_refine, conditioned on the predictor rollout). After several alternations between rollout and denoising, a lightweight Action Head decodes the final action state into a trajectory.

Training is staged: (1) large-scale driving-video pretraining of the vision encoder on OpenScene, DrivingDojo, CoVLA; (2) Auto-Encoder Resampler training on the same corpora; (3) World Predictor training on nuScenes / NAVSIM; (4) joint world–action training that initializes the predictor from stage 3 and trains predictor + denoiser + action head together. Main experiments use 256×512 inputs, ablations 256×256; 150 epochs on 80× A100 with bf16 mixed precision.

Results

NAVSIM v1 (perception-free, Camera-only):

DAWN PDMS 89.1 vs Drive-JEPA 89.0, Epona 86.2, World4Drive 85.1, LAW 83.8 [Table 1].
DAWN best in perception-free NC 98.7, EP 84.3, TTC 96.0; matches C 100; tied/near-best DAC 95.9 [Table 1].
Lower-resolution variant DAWN* hits PDMS 87.9 at 256² — higher resolution adds +1.2 PDMS.

nuScenes end-to-end planning:

L2 (m): 1s 0.17, 2s 0.31, 3s 0.52, Avg 0.33 — best at every horizon. Next-best WorldRFT averages 0.47 m, World4Drive 0.50 m [Table 2].
Collision rate (%): Avg 0.11 vs WorldRFT 0.15, World4Drive 0.16, BEV-Planner 0.59 [Table 2].

Ablations:

WAIM gain: Resampler 82.9 → +World Predictor 85.2 → +Interactive 87.9 PDMS [Table 8].
Coupling-direction ablations confirm both World→Action and Action→World are necessary (drops of −6.3 and −3.0 PDMS respectively) [Table 5].
Interactive rounds saturate at 4; further rounds slightly degrade [Fig. 3].
Rollout horizon: 0s → 2s captures most of the gain (82.8 → 87.3); 4s adds +0.6 PDMS at 1067 ms latency vs 690 ms [Table 6]. Without interaction, the same rollout horizons only reach 85.2 PDMS, isolating the contribution of the interactive use of the rollout.
Resampler token count: 16 → 64 tokens improves PDMS 82.8 → 83.2 but tripled latency (331 ms → 964 ms) [Table 4].

Why it’s interesting

DAWN sits inside the World Foundation Models cluster as a concrete, narrow-domain answer to one of Sitzmann’s open chicken-and-egg sub-problems: how to train action-conditional world models when action–observation paired data is scarce except in self-driving. It also gives the wiki its cleanest filed instance of facebookresearch/vjepa2 — official PyTorch code and checkpoints for V-JEPA 2 and V-JEPA 2.1 being reused as a downstream perception prior — the same predictive-latent backbone that Inference-time Physics Alignment of Video Generative Models with Latent World Models uses as a frozen reward and that V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning extends to dense features. The interactive-refinement framing is structurally close to Diffusion Model Predictive Control (D-MPC), which similarly factorizes dynamics and action proposals with a sample-score-rank planner — but DAWN replaces the explicit MPC loop with weight-tied DiT denoising and a learned predictor that responds to the evolving action hypothesis, rather than treating dynamics and action as independently distilled modules.