Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation (PhyGenBench)

TL;DR

PhyGenBench is an ICML 2025 benchmark of 160 carefully crafted T2V prompts spanning 27 distinct physical laws across four domains — mechanics, optics, thermal, material properties — each prompt designed to exhibit a single, observable physical phenomenon (e.g., “iron gently placed on water” should sink). Alongside it, PhyGenEval is a three-stage hierarchical VLM-judge protocol: keyframe phenomenon detection via VQAScore, multi-image physics-order verification with GPT-4o or LLaVA-Interleave, and full-video naturalness scoring with GPT-4o or InternVideo2. Even Gen-3, the best of the eight evaluated T2V models, scores only 0.51 average PCA; open-source models cluster at 0.36–0.45. PhyGenEval’s overall Spearman correlation with human raters is the only metric that meaningfully tracks human judgments (~0.65), where VideoScore, VideoPhy, and DEVIL all systematically misclassify physically implausible smooth-motion videos.

Key claims

PhyGenBench contains 160 prompts across 27 physical laws covering four categories — Mechanics (gravity, buoyancy, solid/atmospheric pressure, elasticity, friction, surface tension), Optics (reflection, refraction, scattering, dispersion, interference, straight-line propagation), Thermal (six phase transitions including solidification, melting, liquefaction, boiling, deposition, sublimation), and Material Properties (color, hardness, solubility, combustibility, flame reaction; plus acidity, redox, dehydrating properties) [§3].
Each prompt is curated to surface a single physical law via a clear, observable phenomenon — e.g., buoyancy via “iron gently placed on water” (should sink), reflection via “kite above a tranquil pond”, hardness via “egg hurled at a rock” (egg breaks, rock intact) [§3 Overview].
Construction pipeline is five-step: Conceptualization (laws from physics textbooks), Prompt Engineering (manual initial T2V prompts), Prompt Augmentation (detail enrichment that does not reveal the expected phenomenon), Diversity Enhancement (GPT-4o object substitution following T2V-CompBench), and Quality Control [§3 Benchmark Construction].
PhyGenEval decomposes evaluation into Semantic Alignment (SA — object + action presence) and Physical Commonsense Alignment (PCA), each on a four-point scale matching the human protocol [§4].
PCA is a three-stage hierarchical VLM-judge: (i) Key Physical Phenomena Detection via VQAScore on a CLIPScore-retrieved keyframe with GPT-4o-generated physics questions, (ii) Physics Order Verification via multi-image QA on first→key, key→last, and first→last frame pairs, (iii) Overall Naturalness Evaluation against a per-prompt GPT-4o-generated rubric on the full video [§4.2].
Order-verification stage uses GPT-4o or LLaVA-Interleave; naturalness stage uses GPT-4o or InternVideo2; the two-evaluator pair is ensembled per stage [§4.2].
Among eight evaluated T2V models, Gen-3 leads with average PCA 0.51, Kling 0.49, Vchitect 2.0 / CogVideoX-5B / Open-Sora V1.2 cluster at 0.44–0.45, Pika 0.44, CogVideoX-2B 0.37, Lavie 0.36 [Table 2 / Leaderboard].
All models perform best in Optics (where pretraining data carries explicit optical knowledge) and worst on dynamic phenomena — Vchitect 2.0 and CogVideoX-5B reach the optics-domain ceiling of closed models, but lag broadly elsewhere [§5 Quantitative Evaluation].
PhyGenEval’s overall Spearman correlation with human raters substantially exceeds VideoScore, VideoPhy, and DEVIL — VideoScore correlates only ~0.32 on PhyGenBench [Table 1, §4.2].
Existing T2V metrics systematically mislabel physically implausible smooth-motion videos: VideoScore / DEVIL pass “egg bounces off rock like rubber” as physically correct; VideoPhy passes “rock floating on water”; none catch domain-specific failures like “copper flame appears red instead of green” [§5 Human Evaluation, Fig. 4].
Author conclusion: scaling up T2V models or applying prompt engineering is insufficient to fix dynamic-physics failures — the gap to a “world simulator” is structural [§1, §5].

Method

PhyGenBench is built by a five-stage curation: domain selection from physics textbooks (Halliday et al.), per-law manual prompt engineering, GPT-4o detail augmentation that avoids revealing the expected outcome, GPT-4o object substitution for diversity, and human quality control checking that the prompt-law-phenomenon triplet is unambiguous and within current T2V semantic capability.

PhyGenEval is a hierarchical evaluator built per prompt. For each (prompt, law) pair, GPT-4o pre-generates: a retrieval prompt used to locate the phenomenon’s keyframe via CLIPScore, a set of physics-related questions for the keyframe stage, three order-verification questions (first→key, key→last, first→last), and a naturalness rubric. At inference time:

Single-image / keyframe detection: CLIPScore picks the keyframe; VQAScore answers the physics questions on the keyframe and adjacent frames, producing s₁.
Multi-image / order verification: GPT-4o (closed) or LLaVA-Interleave (open) answers the three order questions over the located keyframe pair; s₂ is the mean of the three QA scores.
Full-video naturalness: GPT-4o (closed) or InternVideo2 (open) scores the full video against the GPT-4o-generated per-prompt rubric plus a DEVIL-style global rubric, producing s₃.

The three scores are discretized to 0–3, averaged, and floor-rounded; closed-evaluator (GPT-4o ensemble) and open-evaluator (VQAScore + LLaVA-Interleave + InternVideo2) results are themselves ensembled for the headline score. The whole pipeline runs on a single A100-80G.

Results

Headline: Gen-3 average PCA 0.51 (best); Kling 0.49; Vchitect 2.0 and CogVideoX-5B at 0.45; CogVideoX-2B 0.37 [Table 2].
Per-domain ceiling: Optics is best across the board (Kling 0.58, Gen-3 0.57); Mechanics is hardest (no model exceeds 0.45) [Table 2].
Gen-3 dominates Material Properties (0.51) but ties Kling on Mechanics (0.45) [Table 2].
Among open models, Vchitect 2.0 and CogVideoX-5B beat Pika overall; Lavie lags every category [Table 2, §5].
PhyGenEval overall correlation with human raters substantially higher than competing metrics; VideoScore reaches only ~0.32 on PhyGenBench [Table 1, §4.2].
Failure-mode case studies: VideoScore/DEVIL pass elasticity-violating “egg bounces off rock” videos; VideoPhy passes buoyancy-violating “rock floats” videos; none catch the chemistry-specific “copper flame is green” [§5, Fig. 4].
Cost: PhyGenEval is roughly O(prompt count × VLM calls per stage) — three GPT-4o calls per video plus VQAScore retrieval; closed-source-only evaluation requires only API keys, making the benchmark cheap to run on new models.

Why it’s interesting

PhyGenBench is the October 2024 foundation that the wiki’s more recent physical-realism benchmarks build on or react to. Physion-Eval: Evaluating Physical Realism in Generated Video via Human Reasoning (Mar 2026) explicitly cites the PhyGen line as predecessor and pushes in a different direction — human reasoning traces with timestamped glitch annotations rather than VLM-judge scoring — and reports that 83–94% of frontier 2026 models (Sora 2, Veo 3.1, Kling 2.5) still fail physics-critical scenarios, which is consistent with PhyGenBench’s “even Gen-3 only hits 0.51” finding extending forward through two model generations. PICABench: How Far Are We from Physically Realistic Image Editing? is the explicit image-side analogue, also organized by Optics / Mechanics / etc. PhyGenEval’s three-stage hierarchical VLM-judge protocol is the cleanest filed example of the “decompose physical reasoning into temporally local checks before judging” pattern, which sits between VLM-as-Evaluator‘s static-rubric and dynamic-rubric endpoints — PhyGenEval uses GPT-4o to generate the per-prompt rubric at curation time but applies it frozen at evaluation time. It also pre-figures Inference-time Physics Alignment of Video Generative Models with Latent World Models‘s finding that VLM-judges underperform on physics: PhyGenBench shows VideoScore and DEVIL fail on smooth-motion-implausible-physics inputs, and the inference-time-physics-alignment paper shows the same VLM-judge weakness as a reward signal. For Luma: this is one of the most-cited (ICML 2025) T2V physics benchmarks, with public prompts (160), public evaluation code, public leaderboard, and a working closed-source-only configuration — easy to run on internal models to anchor against Kling/Gen-3 numbers, and the prompt set is small enough to use as a physics-targeted eval gate without dominating CI cost.