PhiZero: A World Model Built Around Physical Language

TL;DR

PhiZero is a video world model that inserts a compact discrete “physical language” between visual state and future frames, so the model first reasons about how the world will change as a token sequence and only then renders those transitions to pixels. The tokenizer is a transition-level Q-Former + FSQ trained self-supervised on 10K hours of unlabeled real-world video, encoding a 4-second clip in 256 discrete symbols vs. Wan2.2 VAE’s 44,800 continuous tokens. A Qwen3-VL-4B-initialized autoregressive VLM predicts the physical-language sequence from the first frame plus a text action intent, and a diffusion prior decodes it back to video. Because the representation is transition-only, the same predicted sequence can be re-rendered with a different first frame, giving zero-shot motion transfer across objects, embodiments, and sim-to-real.

Key claims

• Physical language — a Q-Former + FSQ tokenization of adjacent latent video states — is a much more compact carrier of world dynamics than continuous video latents: 256 discrete symbols per 4-second 33-frame clip vs. 44,800 continuous tokens for the Wan2.2 VAE, while retaining strong reconstruction among highly compressed tokenizers [Abstract, project page tokenization section].
• A reason-then-render decomposition — autoregressive VLM predicts the physical-language sequence, diffusion decoder renders it into video — outperforms end-to-end pixel-space video world models on physics-focused metrics: IQ-Score 41.2 vs. Cosmos3-Super 39.5 / Hunyuan-Video 33.4 / Sora 2 26.5 [Project page Table “IQ-Score”].
• On physics-adherence metrics PhiZero also leads: Physics Score 3.01 vs. Wan2.2-14B 2.90 / Veo3.1 2.85, and Physics Adherence 4.88 vs. Wan2.2-5B 4.51 / Runway 4.27 [Project page Physics tables].
• On the IntPhys2 intuitive-physics understanding benchmark, PhiZero overall = 56.34 beats Gemini-2.5 Flash 55.63, GPT-4o 53.75, and V-JEPA 53.75, showing the physical-language representation transfers to physics understanding, not only generation [Project page IntPhys2 table].
• Because physical language describes state transitions rather than appearance, the same token sequence can be re-rendered with a new first frame to produce zero-shot cross-embodiment transfer (Human → G1 humanoid, Human hand → Sharpa dexterous hand) and sim-to-real transfer without retraining the reasoner [Project page “cross-embodiment” and “sim-to-real” sections].
• Data pipeline: progressive filtering turns a 50K-hour in-the-wild video pool into 10K hours for tokenizer pretraining, 5M four-second clips for tokenizer SFT and reasoner pretraining, and 1M motion-rich physically informative clips for reasoner SFT [Project page data section].

Method

Two-stage architecture. Stage 1 (physical-language tokenizer): a transition-level Q-Former ingests adjacent latent video states (from a pretrained video VAE) and produces an ordered set of query features; FSQ then discretizes them into a physical-language sequence. A diffusion-prior decoder is trained to reconstruct the full video from that sequence conditioned on the clean first frame, forcing the discrete symbols to carry the transition information the frame does not. Stage 2 (reasoner): a Qwen3-VL-4B model is post-trained to autoregressively predict the physical- language sequence given the first frame and a textual action intent; at inference the trained diffusion decoder renders the predicted sequence into video. Because the frame and the transition are decoded separately, editing the first frame while re-using the predicted sequence gives motion transfer across objects, scenes, and embodiments — the paper’s headline capability.

Results

• Video generation quality (IQ-Score): PhiZero 41.2 tops Cosmos3-Super 39.5, Grok-Video 34.8, Hunyuan-Video 33.4, Wan2.2-14B 32.2, Cosmos3-Nano 29.1, Sora 2 26.5, Wan2.2-5B 21.2 [Table “IQ-Score”].
• Physics adherence: Physics Score 3.01 (vs. Wan2.2-14B 2.90, Veo3.1 2.85, OmniWeaving 2.78) and Physics Adherence 4.88 (vs. Wan2.2-5B 4.51, Runway 4.27, Luma 4.13, Mochi 4.14) [Physics tables].
• Intuitive physics understanding (IntPhys2): 56.34 vs. Gemini-2.5 Flash 55.63, VideoMAEv2 / V-JEPA / GPT-4o all 53.75, Cosmos-4B 49.41 [IntPhys2 table].
• Compression: 256 discrete symbols per 4s / 33 frames encode what Wan2.2 VAE stores in 44,800 continuous tokens — ~175× compression while keeping competitive reconstruction [Project page tokenization section].
• Zero-shot transfer: demonstrated on Human → G1 humanoid, Human hand → Sharpa dexterous hand, and Simulation → reality — same physical-language sequence, different first frame, no retraining of the reasoner [Project page transfer section].

Why it’s interesting

PhiZero is the sharpest instantiation to date of “reason-then-render” for video world models. It sits alongside Apple-π: Benchmarking Thinking with Video Towards Law-Grounded Physical Intelligence (Apple-π’s thinking-with-video benchmarks) but flips the polarity: Apple-π asks whether models can think through video, PhiZero argues you get more traction by thinking in a discrete non-pixel representation derived from video and only rendering pixels at the end — a bet that pixel-space video prediction leaves world dynamics implicit and hard to reason over. The tokenizer story also complements VideoRAE: Taming Video Foundation Models for Generative Modeling via Representation Autoencoders (VideoRAE’s representation-autoencoders for continuous video) by pushing to the opposite extreme (FSQ discrete, 175× compression) while retaining physics fidelity, and the zero-shot cross-embodiment transfer demo directly targets territory being staked out by Xiaomi-Robotics-U0: Unified Embodied Synthesis with World Foundation Model and world-action models like 1XWM: From Video to Action — 1X's video-pretrained world model as a NEO robot policy that currently rely on paired action-conditioned data rather than a transferable transition token. If physical-language sequences generalize as the project page claims, PhiZero is the first hint that video world models could publish “action language” the way today’s speech pipelines publish phoneme sequences.

See also

• World Foundation Models — reason-then-render is a distinct WFM design axis vs. end-to-end pixel prediction
• Thinking with Modalities — the CoT here is a learned discrete non-text modality, not text and not pixels
• Autoregressive Video Generation — reasoner is an AR VLM over discrete transition tokens, not pixels
• Representation Autoencoders — physical-language tokenizer is a discrete RAE for state transitions
• Video Generation Benchmarks — new leader on IQ-Score, Physics Score, Physics Adherence, IntPhys2
• Human-to-Robot Retargeting — Human → G1 and Human hand → Sharpa demos via first-frame swap
• Synthetic Training Data — 50K → 10K → 5M → 1M progressive filtering pipeline for tokenizer + reasoner
• 4D Scene Generation — transition tokens generalize across scenes/objects/embodiments
• Apple-π: Benchmarking Thinking with Video Towards Law-Grounded Physical Intelligence — Apple-π’s thinking-with-video benchmarks, contrast in “what does thinking-in-video mean”
• VideoRAE: Taming Video Foundation Models for Generative Modeling via Representation Autoencoders — VideoRAE’s continuous representation autoencoders for video, contrast with discrete tokenization
• Xiaomi-Robotics-U0: Unified Embodied Synthesis with World Foundation Model — XR-U0’s unified WFM for embodied policies, contrast with PhiZero’s action-agnostic transition tokens
• 1XWM: From Video to Action — 1X's video-pretrained world model as a NEO robot policy — 1XWM’s video-to-action world model, contrast in whether cross-embodiment needs paired action data