Skip to content
Bud's Wiki

Video Generation Benchmarks

TL;DR

Section titled “TL;DR”

Benchmarks that score text- or image-conditioned video generators on capabilities beyond raw visual quality — physical commonsense, implicit world rules, robot-task correctness, embodiment behavior, reasoning. The filed papers share a recurring design: a curated prompt set organized by category, an automatic evaluator (typically a VLM- or LMM-as-judge pipeline) calibrated against human raters, and a two-axis report that explicitly separates surface fidelity (VQ, naturalness) from the deeper capability being probed.

Key claims

Section titled “Key claims”

Recent contributions

Section titled “Recent contributions”
  • [2026-08-29] PAWBench: How Far Are We from Probabilistically Aligned World Modeling?: PAWBench (Pu, Zhuo, Paul et al.) opens a distribution-level axis on the cluster distinct from every prior entry (RISE-Video, PhyGenBench, VideoPhy-2, Physics-IQ, Physion-Eval, RBench, Apple-π, Dream.exe, Physion-Arc, PointMotionBench, Spotlight, WMBench, WorldModelGym, HarnessEval-W): formalizes probabilistic alignment as reproducing the correct distribution of possible physical behaviors under fixed initial observation + action, and scores video generators as stochastic samplers via PAWEval (repeated rollouts → empirical outcome distributions vs a reference). Across 50 scenarios × 11 systems, no model simultaneously covers the valid-outcome support and matches reference probabilities; intervention study probes language prompts, initial-noise sampling, and training as candidate levers, none of which closes the gap alone. Missing evaluation counterpart to FlowWM’s training-side “world models should be trained as stochastic samplers” argument.
  • [2026-08-18] HarnessEval-W: Agentifying the Evaluation of Visual Worlds: HarnessEval-W (MirroS + collaborators) opens an agentic evidence-tree axis for the cluster distinct from static-rubric (PhyGenBench, VideoPhy-2), one-shot LMM-judge (RISE-Video), and monolithic automatic pipelines (RBench): a parent agent decomposes each case into subproblems and spawns tool-equipped sub-agents whose verified evidence becomes the audit trail justifying the final verdict. Applied to 18 world models over 330 cases with judgments reported as human-aligned; released as a live community-extensible benchmark (harness + skill libraries + cases + results, code on github.com/mirros-lab/harnesseval-w) — sibling to WorldModelGym and WMBench as filed community-extensible WFM eval platforms.
  • [2026-08-14] H2R-Bench: Benchmarking Human-to-Robot Manipulation Video Generation in World Models: H2R-Bench (SJTU + collaborators) opens a cross-embodiment human-to-robot video-to-video transfer axis not previously covered by the cluster — 11 SOTA video generators × 6 manipulation families × 2 robot embodiments, evaluated on five dimensions (goal-state completion, action-event completion, functional contact transfer, embodiment correctness, general video quality) against source-grounded annotations rather than holistic VLM judgment. Sibling to RBench (Rethinking Video Generation Model for the Embodied World (ReVidGen / RBench / RoVid-X)) at the robot-video-generation level but changes the input channel from text/image to a human demo, and to WMBench (GigaWorld-1: A Roadmap to Build World Models for Robot Policy Evaluation) at the policy-evaluation level. Reports that leading models routinely fail on embodiment consistency, functional interaction, and task execution.
  • [2026-08-06] CineTechBench: A Benchmark for Cinematographic Technique Understanding and Generation: CineTechBench (Beijing UPT) adds a dual understanding+generation film-craft benchmark axis: 610 image QAs + 128 video QAs across 7 cinematography dimensions (scale/angle/composition/movement/lighting/color/focal-length) plus camera-movement I2V reconstruction scored via MonST3R trajectory error. Sibling to FilmBench (which scores narrative multi-shot T2V/R2V) and Physion-Arc (agentic minute-long) at a more diagnostic level — probes fine-grained primitives rather than end-to-end craft. Headline: best MLLM only 70% on stills / 57% on camera-movement videos; I2V rotation error scales sharply with clip angular velocity; even Kling v1.6 gets roll direction wrong.
  • [2026-08-03] Robot-eval-bench: Benchmarking Task Success Evaluation for VLMs: Kite ML’s Robot-eval-bench isolates a keyframes-vs-native-video input-form axis for VLM-judge benchmarks and finds accuracy inverts by family — Gemini judges lose 25-30 points going from four keyframes to full video, while reasoning models stay steady or improve. Also documents that public LeRobot datasets ship some wrong task labels (aloha_static_towel labels a paper-towel-placement clip as “cleaning up a spill”), which corrupts any benchmark grading against shipped labels.
  • [2026-07-31] PhiZero: A World Model Built Around Physical Language: PhiZero reports SOTA on IQ-Score (41.2 vs. Cosmos3-Super 39.5, Sora 2 26.5), Physics Score (3.01 vs. Veo3.1 2.85), Physics Adherence (4.88 vs. Wan2.2-5B 4.51), and IntPhys2 overall (56.34 vs. Gemini-2.5 Flash 55.63), giving one system that leads across generation-quality and physics-understanding benchmarks.
  • [2026-07-31] Towards Video World Models: Xun Huang position essay names physical accuracy as one of five required video-WM properties and cites Kang et al.’s PhyWorld (2411.02385 — not filed) as the sharpest existing demonstration that current video generators achieve combinatorial generalization but fail on extrapolative physics (out-of-distribution velocity/mass). Notes the mitigation direction: dataset curation for rare/OOD events (Ctrl-Crash for car accidents, PISA for object drops) and auxiliary objectives beyond pixel reconstruction (VideoJAM’s joint optical-flow denoising). Complements RISE-Video / Apple-π / PhyGenBench as the concept-level rationale for why physics-focused benchmarks matter.
  • [2026-07-29] FilmBench: A Film-Grade Benchmark for Cinematic Video Generation: FilmBench (Beijing Film Academy + Hujing Digital Media) opens the film-grade cinematic language axis of the cluster, sibling to Physion-Arc but at a different level of abstraction — 1,169 prompts (90.3% multi-shot) reverse-engineered from award-winning films across 20 genres, with a three-level taxonomy co-designed with directors (3 axes × 12 components × 35 T2V + 3 R2V-only sub-metrics) and an open-sourced Cinematic Language operator suite (FilmOps). Auto-evaluator matches human rankings at Spearman ρ = 0.95 (T2V, 9 models) and 0.96 (R2V, 7 models) — matching RBench’s best-in-cluster calibration. Absolute scores fall well below prior web-style benchmarks with consistent dynamic-aesthetics gaps and a single-to-multi-shot drop that widens for weaker models. Distinct from Physion-Arc by scoring underlying T2V/R2V generators on shorter multi-shot prompts rather than end-to-end agentic systems on minute-long screenplays.
  • [2026-07-29] WorldModelGym: a decision-based fidelity benchmark for world models: WorldModelGym (Reka Labs, 2026-07-21 blog + leaderboard live 2026-07-28) adds the decision-based fidelity axis to the cluster — distinct from every prior entry (RISE-Video, PhyGenBench, VideoPhy-2, Physics-IQ, RBench, Apple-π, Dream.exe, Physion-Arc, PointMotionBench, Spotlight, MMBench2, WMBench). 100+ tracks across Atari, Meta-World, DMC, and classic control; models rank five pre-computed action sequences under a fixed argmax rule, scored by normalized regret and reward-prediction error (the latter separates “picks well” from “predicts well”). Deliberately open-loop and multiple-choice rather than full-rollout, so compute-heavy pixel WFMs stay tractable; state representation is opaque so JEPA, token, and pixel WFMs all submit through the same Gymnasium adapter.
  • [2026-07-28] GigaWorld-1: A Roadmap to Build World Models for Robot Policy Evaluation: WMBench adds a policy-evaluation-oriented axis to the cluster distinct from every previously filed entry (RISE-Video, PhyGenBench, VideoPhy-2, Physics-IQ, RBench, Apple-π, Dream.exe, Physion-Arc, PointMotionBench, Spotlight, MMBench2): 324K+ matched real-robot / simulated policy rollouts across diverse manipulation tasks, controlled over 7 WFMs × 4 action encodings × rollout horizons × evaluation metrics. Sharpens the “surface fidelity vs deeper capability” split the cluster tracks by empirically claiming that short-term visual realism (what most cluster entries probe) is not the axis that separates good policy evaluators from bad ones — long-horizon action-faithful rollout consistency is.
  • [2026-07-22] Physion-Arc 1.0: Benchmarking Video Agents on Minute-Long Video Generation: Physion-Arc 1.0 (Physion Labs) opens the narrative/cinematic axis of the cluster — six agentic video systems (Runway Agent 2.0, Luma, MiniMax, Kling, Utopai, TapNow) scored on 100 screenplays / 600 minute-long, multi-shot videos across narrative coherence, cinematic language, pacing, direction, emotional impact, and production quality. Distinct from every previously filed entry (PhyGenBench, VideoPhy-2, Physics-IQ, RBench, RISE-Video, Apple-π, Dream.exe, MMBench2, Spotlight, PointMotionBench) by targeting minute-long multi-shot narrative output from agents rather than 5-10s clips from single-model generators. Runway Agent 2.0 leads all eight announced dimensions, with the largest margin on subjective/cinematic-taste dimensions. Blog URL 404s at filing time; details from the PhysionLabs announcement and companion Physion-Eval HF dataset.
  • [2026-07-21] Apple-π: Benchmarking Thinking with Video Towards Law-Grounded Physical Intelligence: Apple-π (Yao, Hu, Cao et al. — MMLab@NTU) adds a law-grounded stage-decomposed axis to the cluster: 400 videos across 10 classical-mechanics tasks with a three-stage Perception → Formulation → Deduction protocol using chain-of-frames prompting on infographic-annotated first frames, and a hybrid MLLM-subjective + physics-law-grounded-objective evaluation suite. Headline: best of 11 evaluated video models scores only 0.473, with an explicit stage decomposition showing models can perceive physics but fail to formulate laws and deduce dynamics — plus a persistent Sim-to-Real gap and weak multi-law transfer. Complements PhyGenBench (law-taxonomized VLM-judged commonsense), VideoPhy-2 (action-centric), and Physics-IQ (real-world filmed pixel-mask) by anchoring the objective in explicit law computation rather than VLM naturalness judgment.
  • [2026-07-14] Towards Generalizable Robotic Manipulation in Dynamic Environments (DOMINO / PUMA): DOMINO (Fang et al., ECCV 2026) is the dynamic-manipulation counterpart to Dream.exe on this cluster’s embodied-execution axis: 35 tasks × 3 dynamic-complexity levels × 110K+ trajectories on RoboTwin 2.0, scored via Success Rate + a Manipulation Score that penalizes collisions and out-of-bounds. Where Dream.exe evaluates whether video generators produce executable robot motion, DOMINO evaluates whether VLA policies survive scene-native motion — third-party numbers (InternVLA-A1.5 SOTA at 27.7 vs its 98.9 on LIBERO) confirm it opens a large capability gap invisible to quasi-static suites.
  • [2026-06-29] PhysisForcing: Physics Reinforced World Simulator for Robotic Manipulation: PhysisForcing exercises R-Bench, PAI-Bench, and EZS-Bench jointly as a robot-oriented evaluation suite and reports first explicit training-time intervention against the physical-instability failure modes the benchmarks diagnose.
  • [2026-06-27] Hallucination in World Models is Predictable and Preventable (MMBench2): MMBench2 (Hansen & Wang) adds a benchmark-with-a-reference-model axis: 210 tasks × 10 domains with live simulators, ground-truth actions/rewards/language and a 350M Dreamer-4-style world model trained on it — distinct from the cluster’s other entries (RISE-Video, PhyGenBench, VideoPhy-2, Physics-IQ, RBench, Dream.exe) which evaluate external models on held-out video sets; here the benchmark, the trained model, three label-free runtime hallucination predictors (ρ ≈ 0.8 vs realized rollout error), and a coverage-targeted mitigation all ship together.
  • [2026-06-20] Spotlight: Identifying and Localizing Video Generation Errors Using VLMs: Spotlight adds the fine-grained temporal-localization axis to the cluster — 600 Veo 3 / Seedance / LTX-2 videos with 1,604 human-annotated errors (six categories, start/end times, severity 1-5, natural-language reasons); humans hit S+P = 0.508 vs best VLM 0.254, and different VLM judges disagree on which generator is worst, making it a useful sanity check on leaderboards built from single-judge holistic scores.
  • [2026-06-19] MolmoMotion: Language-guided 3D motion forecasting: PointMotionBench (2.7K human-validated clips, 111 object categories × 61 motion types, spanning indoor manipulation, egocentric HOI, and outdoor dynamic scenes) directly evaluates 3D motion forecasting via point-trajectory ADE — a non-VLM-judge benchmark complement to RISE-Video / PhyGenBench that measures whether generators produce physically correct 3D motion rather than whether a VLM thinks the output looks plausible.
  • [2026-06-11] Do generative video models understand physical principles? (Physics-IQ): Physics-IQ adds the real-world filmed axis to the cluster — 66 staged physical scenarios × 3 perspectives × 2 takes = 396 8-second videos, with a predict-the-5-second-continuation protocol that compares model motion masks against ground-truth motion masks via four pixel-level metrics (Spatial IoU, Spatiotemporal IoU, Weighted Spatial IoU, MSE) normalized so the take-1-vs-take-2 physical-variance ceiling is 100. Headline: best model (VideoPoet multiframe) scores 24.1, Sora last at 8.7, and a Gemini-1.5-Pro 2AFC realism check shows realism and physics are cross-model uncorrelated.
  • [2026-06-06] Dream.exe: Can Video Generation Models Dream Executable Robot Manipulation?: Dream.exe (showlab) adds an embodied-execution axis to the cluster — 101 manipulation tasks across three physical-complexity levels, 8 generators (frontier closed-source, open-source, robotics-specific), scored on visual quality + trajectory fidelity + simulator execution success; the headline finding is that visual quality does not predict executability.
  • [2026-06-03] VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generation: VideoPhy-2 adds the action-centric axis to the cluster — 197 real-world actions × ~3 prompts × ~70K human annotations, with strict SA/PC separation (rated as independent tasks to avoid prompt-conditioning bias), per-video physical-rule labels, and a model-built hard subset of 89 prompts. Reports Wan2.1-14B 32.6% joint full / 21.9% hard with closed models (Sora, Ray2) trailing the top open models. Also releases VideoPhy-2-Autoeval (7B, distilled from ~50K human ratings) beating Gemini-2.0-Flash-Exp by large relative margins on SA/PC correlation and rule classification.
  • [2026-06-03] Rethinking Video Generation Model for the Embodied World (ReVidGen / RBench / RoVid-X): RBench introduces the robot-oriented axis to the cluster — 25 models scored across 5 task domains × 4 embodiments with reproducible automatic sub-metrics (structural consistency, physical plausibility, action completeness). Achieves the strongest filed automatic-vs-human alignment (Spearman 0.96) of any cluster entry, and surfaces a sharp result that robotics-specific models trail general commercial video models on robotics evaluation.
  • [2026-05-24] Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation (PhyGenBench): PhyGenBench (ICML 2025) anchors the physical-commonsense axis — 160 T2V prompts spanning 27 physical laws across Mechanics / Optics / Thermal / Material Properties, with the three-stage PhyGenEval VLM-judge protocol (keyframe → multi-image order → full-video naturalness).
  • [2026-05-23] RISE-Video: Can Video Generators Decode Implicit World Rules?: RISE-Video introduces the implicit-world-rules axis for TI2V — 467 samples × 8 knowledge categories × 4 metrics (RA 0.4 / TC 0.25 / PR 0.25 / VQ 0.1) with a relaxed weighted score and a strict per-sample accuracy that exposes a 1.9–22.5% accuracy band hidden by 74.5–96.2% VQ.

Open questions

Section titled “Open questions”
  • Do these benchmarks correlate with each other on overlapping models? PhyGenBench, VideoPhy-2, and Physics-IQ all score Sora and Gen-3-class models — Physics-IQ ranks Sora last on physics but first on visual realism; PhyGenBench ranks Gen-3 first; VideoPhy-2 has Sora trailing Wan2.1-14B. No filed paper cross-validates rankings across the three.
  • Is the LMM-as-judge pipeline at the core of each benchmark itself validated against human raters at absolute accuracy (not just rank correlation)? RBench’s Spearman 0.96 is strong but methodology details for the auto-evaluator are pending arxiv fetch.
  • The “robotics-specific models trail general models” finding from RBench is striking — is this an evaluation artifact (general models look better on RBench’s chosen metrics) or a real capability gap? No filed paper triangulates Cosmos / DreamGen against Wan/Seedance on a non-RBench robotics task.
  • The “closed models don’t beat open” finding on VideoPhy-2 and Physics-IQ — does it survive when newer closed flagships (Sora 2, Veo 3, Kling 2.6) are run through the same protocol? Physion-Eval (Physion-Eval: Evaluating Physical Realism in Generated Video via Human Reasoning) tested those models but with a different metric.
  • How well do these benchmarks generalize to closed flagship products (Genie 3, Sora 2)? Genie 3 is absent from all filed leaderboards; only VideoPhy-2 (original Sora), Physics-IQ (original Sora), and RBench (Wan 2.6, Seedance 1.5 Pro) include closed-flagship entries.
  • Should there be a fourth cluster axis for temporal/motion benchmarks (Pulse of Motion, HumanScore, NewtonGen) — currently those sit under World Foundation Models and VLM-as-Evaluator but not under a dedicated benchmark concept?
  • Distilled 7B auto-evaluators (VideoPhy-2-Autoeval) beat much larger frozen VLM judges (Gemini-2.0-Flash-Exp). Is this a general pattern across the cluster, or specific to physics-commonsense?
  • Physics-IQ avoids VLM-judge weaknesses entirely by comparing pixel motion masks; PhyGenBench / VideoPhy-2 lean on VLM judges that themselves fail on physics. Is motion-mask comparison the right primitive when models eventually stop having static-camera mode (e.g., Sora-style autonomous cuts)?

Papers

Section titled “Papers”

2026-01

Section titled “2026-01”

2026-02

Section titled “2026-02”

2025-03

Section titled “2025-03”

2025-01

Section titled “2025-01”

2024-10

Section titled “2024-10”