PAN: A World Model for General, Interactable, and Long-Horizon World Simulation

TL;DR

PAN is MBZUAI IFM’s open general world model: an LLM-backed autoregressive latent dynamics module (Qwen2.5-VL-7B-Instruct) paired with a video diffusion decoder (Wan2.1-T2V-14B) that predicts the next latent world state from history + a natural-language action, then decodes it into a temporally coherent video chunk. The team instantiates a Generative Latent Prediction (GLP) architecture where latent dynamics are grounded in observation reconstruction (rejecting JEPA-style latent-matching as collapse-prone), and introduces Causal Swin-DPM — a sliding-window denoising scheme with chunk-wise causal attention — to suppress error accumulation across long rollouts. Trained on large-scale video-action pairs spanning diverse embodied and physical domains, PAN reports strong open-source results on action-conditioned simulation, long-horizon forecasting, and simulative reasoning.

Key claims

A general world model must unify two abilities: simulating future world evolutions via internal latent dynamics, and grounding those dynamics in observable sensory experience — GLP formalizes this with an encoder, a predictive module, and a decoder defining a generative process over the next observation [§2.1].
JEPA-style latent-matching objectives are prone to collapse and “indefinability” — the encoder/predictor can trivially minimize loss by mapping all observations to a constant; PAN’s generative supervision in observation space avoids this by anchoring each latent transition to a realizable sensory change [§2.2].
PAN parameterizes the GLP encoder + backbone with a single VLM (Qwen2.5-VL-7B-Instruct) — the vision tower produces structured spatial embeddings as the world state, and the language backbone predicts the next latent state given the multi-turn history of states and natural-language actions, alternating in a conversational format with 256 learnable query tokens per step [§3.1, §3.2, Fig. 1].
The video diffusion decoder is adapted from Wan2.1-T2V-14B and trained with a flow-matching objective (Rectified Flow, 1000 denoising steps, shifted schedule) to reconstruct the next video chunk conditioned on the predicted latent and the most recent observation [§3.3.1].
Naively chaining single-shot video diffusion by conditioning only on the last frame causes (a) local inconsistency at chunk boundaries and (b) rapid quality degradation from error accumulation over long rollouts [§3.3.3].
Causal Swin-DPM addresses both: a sliding temporal window simultaneously holds two chunks at different noise levels (earlier chunk at noise τ/2, later chunk at full noise τ), and after τ/2 denoising steps the earlier chunk is dequeued and a new noisy chunk is enqueued; chunk-wise causal attention masks future-chunk actions to preserve real-time interactivity [§3.3.3, Fig. 3].
Conditioning the future chunk on a partially noised (rather than fully denoised) history is itself a feature, not a bug: it suppresses incidental pixel-level details that are not reliably predictive (e.g. occluded sides of objects) and emphasizes persistent semantic consistency, so the model is not penalized for inherently unknowable details [§3.3.3].
PAN uses the Wan2.1 3D causal VAE for frame compression, operating in VAE latent space with a 21-token window corresponding to 81 real video frames [§3.3.4].
PAN claims state-of-the-art performance among open-source systems on action-conditioned simulation, long-horizon forecasting, and simulative reasoning, and “within range of leading commercial models” [Abstract, §1].

Method

PAN instantiates the Generative Latent Prediction (GLP) architecture with three components:

Vision encoder — the vision tower of Qwen2.5-VL-7B-Instruct, which partitions each frame into patches, applies windowed self-attention with 2D RoPE, and uses 3D patch partitioning for short-term motion. Output: structured visual tokens that serve as the latent world state.
Autoregressive world model backbone — the Qwen2.5-VL-7B-Instruct language model rolls forward state given a multi-turn (<state>, <action>, <state>, …) chat-format history plus 256 learnable query embeddings. Teacher-forced at training, closed-loop at inference. Output: 256 continuous tokens encoding dynamics and object-agent interactions.
Video diffusion decoder — Wan2.1-T2V-14B DiT adapted with Causal Swin-DPM. At each denoising round the temporal window holds two chunks at staggered noise levels (τ/2 and τ); after τ/2 steps the earlier chunk dequeues and a new noisy chunk enqueues. Each chunk conditions on its own action via dedicated cross-attention; a chunk-wise causal attention mask prevents the later chunk from attending to future actions. Decoder uses flow matching with 1000 denoising steps in Rectified Flow form.

Training uses large-scale video-action pairs spanning diverse physical and embodied domains — explicitly motivated by the observation that prior video corpora (short, independent clips) lack action-conditioned temporal continuity. The generative-supervision objective ([Eq. 5]) minimizes the discrepancy between the predicted and ground-truth next observation; each latent transition is anchored to a realizable sensory change.

Results

The paper claims PAN reaches best open-source fidelity for action simulation across both agentic scenarios (a controllable entity must follow instructions without breaking the background) and across long-horizon forecasting and simulative planning [Abstract; §1]. The detailed numerical tables are in §7 (truncated in the fetched copy at filing time); the headline framing is that PAN is “state-of-the-art among open-source systems and within range of leading commercial models” on action-conditioned interaction, in contrast to prompt-to-full-video models which lack interactive controllability. Third-party coverage at the project page reports the same framing without specific scores.

Why it’s interesting

PAN is the clearest published instance to date of the VLM-as-world-model-backbone + diffusion-as-decoder factorization that several filed papers have been gesturing at separately: it combines the “interactive rollout WFM” framing of Project Genie: Experimenting with infinite, interactive worlds / Introducing Runway Labs with an explicit architectural commitment to latent prediction grounded in pixel reconstruction — directly opposing the JEPA-style latent-matching line (Introducing the V-JEPA 2 world model and new benchmarks for physical reasoning, V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning) on grounds the paper argues from collapse-and-indefinability. Causal Swin-DPM is structurally a cousin of Context Forcing: Consistent Autoregressive Video Generation with Long Context‘s drift-suppression machinery and MALT Diffusion: Memory-Augmented Latent Transformers for Any-Length Video Generation‘s noise-augmented memory: all three accept that long-horizon stability requires perturbing the history at training/inference time rather than pretending it’s clean. This is the first filed open-weight GLP-architecture WFM with both a public arxiv report and demos, in contrast to closed flagships like Genie 3.