Physion-Eval: Evaluating Physical Realism in Generated Video via Human Reasoning

TL;DR

Physion-Eval is a benchmark of human expert reasoning traces for diagnosing physical-realism failures in AI-generated videos. It contains 10,990 expert annotations across 12,718 videos from five state-of-the-art generators, covering 22 physical-phenomena categories split into 17 exocentric (dynamics, thermodynamics, optics, sourced from WISA-80K) and 5 egocentric (physical interaction, sourced from EPIC-KITCHENS). Each annotation is temporally localized — a glitch is tied to a timestamp, a structured failure category, and a natural-language explanation of which physical principle was violated. The headline finding: 83.3% of exocentric and 93.5% of egocentric generated videos contain at least one human-identifiable physical glitch, while current MLLM critics largely miss these failures, especially in the egocentric setting.

Key claims

Physion-Eval contains 10,990 expert reasoning traces across 12,718 generated videos from five state-of-the-art video generators (incl. Sora 2, Veo 3.1 Fast, Kling 2.5), spanning egocentric and exocentric views [Abstract, §1].
The taxonomy covers 22 fine-grained physical categories — 17 exocentric (6 dynamics, 6 thermodynamics, 5 optics) inherited from WISA-80K, plus 5 egocentric physical-interaction categories defined for this work [§3.1, footnote 1].
Each generated video is paired with a real-world reference video that depicts a clear physical process — the annotation is grounded by the reference’s behavior, not by abstract physical theory alone [Abstract].
Annotations are temporally localized: each glitch carries a timestamp (e.g. “at 2.9s the jam on the knife tip suddenly disappears”), a structured failure category, and a free-form natural-language explanation [Abstract, dataset preview].
The structured failure categories observable on the dataset include Object Permanence Violation, Temporal Coherence Breakdown, Force & Motion Inconsistency, Material / State Inconsistency, Contact / Interaction Failure, Geometric / Collision Violation, Causal Sequence Violation, and Other Failures [Dataset schema].
In physics-critical scenarios, 83.3% of exocentric and 93.5% of egocentric generated videos exhibit at least one human-identifiable physical glitch [Abstract, §1].
Untrained human viewers reliably detect physical-glitch failures; current MLLM critics largely miss them, with the gap especially large in egocentric settings — a clear separation between human perception and automated evaluation [§5/Conclusion].
This is the first large-scale, human-centered evaluation of physical realism in video generation [§5/Conclusion].

Method

Physion-Eval is built by pairing each generated video with a real-world reference video depicting a clear physical process. Exocentric references come from WISA-80K (the 80K-clip physics-categorized video corpus from WISA); egocentric references come from EPIC-KITCHENS. Five frontier video generators (publicly visible in the data: Sora 2, Veo 3.1 Fast, Kling 2.5, plus two more) are conditioned on text/image derived from each reference and asked to produce the same physical process.

Human expert annotators then watch each generated video against the reference and produce a structured annotation: a binary has_glitches flag, a glitch_severity score (1-5), a Glitch_Category from a fixed taxonomy (Object Permanence Violation / Temporal Coherence Breakdown / Force & Motion Inconsistency / Material / State Inconsistency / Contact / Interaction Failure / Geometric / Collision Violation / Causal Sequence Violation / Other Failures / No Issue), and a free-form reasoning field that timestamps each glitch and explains in natural language which physical principle was violated. The taxonomy itself is split into 22 fine-grained physical categories — 17 exocentric (6 dynamics, 6 thermodynamics, 5 optics) inherited from WISA and 5 egocentric physical-interaction categories defined in this work.

The benchmark is released as a HuggingFace dataset (PhysionLabs/Physion-Eval); a comparison protocol is implied by the structured annotation but the paper itself frames Physion-Eval primarily as a diagnostic resource rather than as a leaderboard with a single headline number.

Results

Exocentric setting: 83.3% of generated videos contain at least one human-identifiable physical glitch.
Egocentric setting: 93.5% — substantially worse than exocentric, consistent with egocentric video being a structurally harder distribution (hand-object contact, occlusion, first-person motion) [Abstract].
Untrained human raters reliably catch these glitches, while frontier MLLM critics miss most of them; the human–MLLM gap is largest in the egocentric subset [§5/Conclusion].
Dataset scale: 10,990 expert reasoning traces across 12,718 generated videos from five generators; per-sample severity ranges 0-5 (visible in the released preview rows).
Within the released preview, severity-5 glitches concentrate in Material/State Inconsistency and Force & Motion Inconsistency categories — e.g. asymmetric container deformation under explosion without external cause, fluid columns lacking surface tension, paper burning from edges with central paper unaffected.

Why it’s interesting

This is the cleanest filed counter-data point to anyone claiming current frontier video models already work as world simulators: even on physics-critical scenarios curated from existing physics-tagged corpora (WISA, EPIC-KITCHENS), Sora 2 / Veo 3.1 / Kling 2.5 fail human inspection 83-94% of the time. The egocentric → exocentric gap (93.5 vs 83.3) is a concrete prediction about where video generators break first, and it lines up with the much earlier failure mode observed in RISE-Video: Can Video Generators Decode Implicit World Rules? — strict reasoning accuracy capped at 22.5% across 11 TI2V models, even when per-frame Visual Quality stays >90%. The MLLM-judge-misses-physics finding sharpens Inference-time Physics Alignment of Video Generative Models with Latent World Models‘s observation that VLM-based rewards (Qwen2.5-VL, Qwen3-VL) regress baselines on PhysicsIQ while VJEPA-2’s predictive surprise works — Physion-Eval supplies the human-annotated ground truth that lets future work calibrate which automated proxies (latent-WFM surprise, VLM-judge, rule-based) actually track human perception of physical realism. Adjacent on the image side: PICABench: How Far Are We from Physically Realistic Image Editing? argues the same gap for instruction-based image editing (only closed-source models clear 60). For Luma: the dataset’s reasoning traces are usable as supervision for physics-aware reward models, not just as a leaderboard, and the author’s offer to privately benchmark Luma’s models is worth taking up — it yields numbers on the same 22-category taxonomy that frontier closed models are scored on.