VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generation

TL;DR

VideoPhy-2 is an action-centric T2V physical commonsense benchmark of 590 prompts across 197 real-world actions (sports, physical activities, object interactions) with ~70K human annotations on a 1-5 scale separating Semantic Adherence (SA) from Physical Commonsense (PC) plus per-video physical-rule labels. The best evaluated model, Wan2.1-14B, reaches only 32.6% joint score (SA≥4 ∧ PC≥4) on the full set and 21.9% on the hard subset; closed models (Sora, Ray2) do not beat the strongest open models. The authors also release VideoPhy-2-Autoeval, a 7B multi-task evaluator distilled from human ratings that substantially outperforms Gemini-2.0-Flash-Exp on SA/PC correlation and on rule-following classification.

Key claims

The dataset contains 590 captions across 197 actions (143 physical/sports, 54 object interactions) with avg original/upsampled caption lengths of 26 / 138 tokens, yielding 6× more captions and 3–5× longer prompts than the original VideoPhy [§2, Table 1].
Actions are selected from Kinetics/UCF-101/SSv2 by two independent STEM-graduate annotator groups for relevance to physical commonsense (gravity, elasticity, buoyancy, reflection, conservation of mass/momentum), then de-duplicated by Gemini-2.0-Flash-Exp [§2 Stage 1].
Physical rules are generated post-hoc from captions of the generated videos (not the input prompts) via Gemini-2.0-Flash-Exp, with three candidate rules per video, and human annotators may add missed violations during evaluation [§2 Stage 3].
A hard subset of 89 prompts is selected by taking actions on which CogVideoX-5B fails joint SA/PC entirely — these concentrate on momentum transfer, state changes, balancing, and complex motions (backflip, pole vault, pizza-tossing) [§2 Stage 4, Appendix Table 10].
SA and PC are scored on independent 5-point scales by separate tasks, deliberately preventing prompt-conditioning bias in PC ratings (a methodological departure from VideoPhy where both axes were rated jointly) [§3.2].
Wan2.1-14B leads with joint score 32.6% on full / 21.9% hard; CogVideoX-5B 25.0% / 0.0%; Cosmos-Diff-7B 24.1% / 10.9%; Sora 23.3% / 5.3%; Ray2 20.3% / 8.3%; HunyuanVideo-13B 17.2%; VideoCrafter-2 10.5% [Table 2].
Closed models (Sora, Ray2) do not outperform the strongest open models (Wan2.1-14B, CogVideoX-5B) on physical commonsense [Table 2, §5.1].
Performance on Physical Activities (sports) is systematically lower than on Object Interactions across all models, suggesting sports-video curation is the missing ingredient [Table 2, §5.1].
PC scores are essentially uncorrelated with aesthetics (r=0.09 vs LAION classifier) and motion-quality metrics (r=0.002 vs RAFT optical flow), so a model cannot reach high VideoPhy-2 PC by optimizing surface metrics [Table 3].
Models fail most often on conservation laws (mass and momentum), per the fine-grained physical-rule annotations [§1, §5.1].
VideoPhy-2-Autoeval (7B, fine-tuned from VideoCon-Physics) beats Gemini-2.0-Flash-Exp by +80.8% / +236.4% relative Pearson on SA/PC for unseen prompts and +45.2% / +281.8% for unseen video models [Table 4].
On rule-classification accuracy, VideoPhy-2-Autoeval reaches 78.7% on unseen prompts vs Gemini-2.0-Flash-Exp 59.2% (+32.9%) [Table 5b].

Method

Four-stage curation: (1) curate 197 actions from Kinetics/UCF-101/SSv2 via two independent STEM-graduate annotation passes + Gemini-2.0-Flash-Exp dedup; (2) for each action, generate 20 prompts with Gemini-2.0-Flash-Exp following four principles (visible interactions, no abstract/sensory detail, diverse characters/objects, multiple events) and upsample with the Mistral-NeMo-12B Cosmos prompt upsampler; (3) generate videos from a reference model, caption them with Gemini-2.0-Flash-Exp, and ask Gemini to write three candidate physical rules per video grounded in the caption; (4) build a hard subset by selecting actions where CogVideoX-5B scores 0 on joint SA/PC.

For evaluation, three AMT annotators rate each video separately on SA (with prompt) and on PC + per-rule grounding (without prompt). The headline joint score thresholds SA≥4 and PC≥4. Per-rule labels are {followed, violated, cannot-be-determined}; annotators may add free-text rule violations that are later normalized into rule statements via Gemini.

VideoPhy-2-Autoeval is a 7B multi-task video-language model: it shares a VideoCon-Physics backbone and is trained on ~50K human annotations across three tasks (SA scoring, PC scoring, rule classification) with task-specific prompt templates, enabling inter-task transfer.

Results

Headline joint (SA≥4 ∧ PC≥4): Wan2.1-14B 32.6% full / 21.9% hard; next-best open is CogVideoX-5B 25.0% / 0.0% [Table 2].
Closed-model comparison: Sora 23.3% / 5.3% and Ray2 20.3% / 8.3% sit below the top open models on both splits [Table 2].
The hard subset induces a ~10–25 pp drop across the leaderboard, validating the model-driven hard-subset construction [Table 2].
Object Interactions outperform Physical Activities for nearly every model — e.g. Wan2.1-14B 36.2% vs 31.5%; Cosmos-Diff-7B 27.4% vs 22.6% [Table 2].
VideoPhy-2-Autoeval Pearson correlation with humans on unseen prompts: SA 0.47 / PC 0.37, vs Gemini-2.0-Flash-Exp 0.26 / 0.11 and VideoCon-Physics 0.32 / 0.25 [Table 4].
Rule-classification accuracy 78.7% (unseen prompts) and 72.9% (unseen video models) vs Gemini 59.2% / 57.1% and random 34.5% / 31.2% [Table 5b].
The benchmark+auto-eval combined human-annotation cost is approximately the K-annotation figures reported across collection and training; auto-evaluator inference runs on a single A100-class GPU.

Why it’s interesting

VideoPhy-2 is the action-centric counterpart to Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation (PhyGenBench): where PhyGenBench enumerates 27 physics laws via single-phenomenon prompts and judges them with the PhyGenEval VLM-cascade, VideoPhy-2 starts from 197 real-world actions, generates rules from the produced videos, and reports joint SA∧PC. The two benchmarks attack the same gap from opposite directions — taxonomy-first vs. action-first — and broadly agree that even frontier T2V models top out far below human commonsense (PhyGenBench’s Gen-3 0.51 PCA mirrors VideoPhy-2’s Wan2.1-14B 32.6% joint). It also extends Physion-Eval: Evaluating Physical Realism in Generated Video via Human Reasoning‘s frontier-models-still-fail finding back one model generation and adds per-rule physical-law grounding rather than free-form glitch reasoning. The PC↔aesthetics decorrelation (r=0.09) is the strongest filed argument against the temptation to use VLM-as-Evaluator aesthetic/motion proxies as physics rewards — a finding consistent with Inference-time Physics Alignment of Video Generative Models with Latent World Models‘s observation that VLM-judge rewards regress on PhysicsIQ. For Luma, this is a useful drop-in eval gate with a small enough prompt set (590) to run as a CI check, a clean SA/PC separation that mirrors internal eval taxonomies, and the explicit “sports activities are the missing data” signal that aligns with action-centric capture needs.