Skip to content
Bud's Wiki

Video Generation Benchmarks

TL;DR

Section titled “TL;DR”

Benchmarks that score text- or image-conditioned video generators on capabilities beyond raw visual quality — physical commonsense, implicit world rules, robot-task correctness, embodiment behavior, reasoning. The filed papers share a recurring design: a curated prompt set organized by category, an automatic evaluator (typically a VLM- or LMM-as-judge pipeline) calibrated against human raters, and a two-axis report that explicitly separates surface fidelity (VQ, naturalness) from the deeper capability being probed.

Key claims

Section titled “Key claims”

Recent contributions

Section titled “Recent contributions”
  • [2026-07-14] Towards Generalizable Robotic Manipulation in Dynamic Environments (DOMINO / PUMA): DOMINO (Fang et al., ECCV 2026) is the dynamic-manipulation counterpart to Dream.exe on this cluster’s embodied-execution axis: 35 tasks × 3 dynamic-complexity levels × 110K+ trajectories on RoboTwin 2.0, scored via Success Rate + a Manipulation Score that penalizes collisions and out-of-bounds. Where Dream.exe evaluates whether video generators produce executable robot motion, DOMINO evaluates whether VLA policies survive scene-native motion — third-party numbers (InternVLA-A1.5 SOTA at 27.7 vs its 98.9 on LIBERO) confirm it opens a large capability gap invisible to quasi-static suites.
  • [2026-06-29] PhysisForcing: Physics Reinforced World Simulator for Robotic Manipulation: PhysisForcing exercises R-Bench, PAI-Bench, and EZS-Bench jointly as a robot-oriented evaluation suite and reports first explicit training-time intervention against the physical-instability failure modes the benchmarks diagnose.
  • [2026-06-27] Hallucination in World Models is Predictable and Preventable (MMBench2): MMBench2 (Hansen & Wang) adds a benchmark-with-a-reference-model axis: 210 tasks × 10 domains with live simulators, ground-truth actions/rewards/language and a 350M Dreamer-4-style world model trained on it — distinct from the cluster’s other entries (RISE-Video, PhyGenBench, VideoPhy-2, Physics-IQ, RBench, Dream.exe) which evaluate external models on held-out video sets; here the benchmark, the trained model, three label-free runtime hallucination predictors (ρ ≈ 0.8 vs realized rollout error), and a coverage-targeted mitigation all ship together.
  • [2026-06-20] Spotlight: Identifying and Localizing Video Generation Errors Using VLMs: Spotlight adds the fine-grained temporal-localization axis to the cluster — 600 Veo 3 / Seedance / LTX-2 videos with 1,604 human-annotated errors (six categories, start/end times, severity 1-5, natural-language reasons); humans hit S+P = 0.508 vs best VLM 0.254, and different VLM judges disagree on which generator is worst, making it a useful sanity check on leaderboards built from single-judge holistic scores.
  • [2026-06-19] MolmoMotion: Language-guided 3D motion forecasting: PointMotionBench (2.7K human-validated clips, 111 object categories × 61 motion types, spanning indoor manipulation, egocentric HOI, and outdoor dynamic scenes) directly evaluates 3D motion forecasting via point-trajectory ADE — a non-VLM-judge benchmark complement to RISE-Video / PhyGenBench that measures whether generators produce physically correct 3D motion rather than whether a VLM thinks the output looks plausible.
  • [2026-06-11] Do generative video models understand physical principles? (Physics-IQ): Physics-IQ adds the real-world filmed axis to the cluster — 66 staged physical scenarios × 3 perspectives × 2 takes = 396 8-second videos, with a predict-the-5-second-continuation protocol that compares model motion masks against ground-truth motion masks via four pixel-level metrics (Spatial IoU, Spatiotemporal IoU, Weighted Spatial IoU, MSE) normalized so the take-1-vs-take-2 physical-variance ceiling is 100. Headline: best model (VideoPoet multiframe) scores 24.1, Sora last at 8.7, and a Gemini-1.5-Pro 2AFC realism check shows realism and physics are cross-model uncorrelated.
  • [2026-06-06] Dream.exe: Can Video Generation Models Dream Executable Robot Manipulation?: Dream.exe (showlab) adds an embodied-execution axis to the cluster — 101 manipulation tasks across three physical-complexity levels, 8 generators (frontier closed-source, open-source, robotics-specific), scored on visual quality + trajectory fidelity + simulator execution success; the headline finding is that visual quality does not predict executability.
  • [2026-06-03] VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generation: VideoPhy-2 adds the action-centric axis to the cluster — 197 real-world actions × ~3 prompts × ~70K human annotations, with strict SA/PC separation (rated as independent tasks to avoid prompt-conditioning bias), per-video physical-rule labels, and a model-built hard subset of 89 prompts. Reports Wan2.1-14B 32.6% joint full / 21.9% hard with closed models (Sora, Ray2) trailing the top open models. Also releases VideoPhy-2-Autoeval (7B, distilled from ~50K human ratings) beating Gemini-2.0-Flash-Exp by large relative margins on SA/PC correlation and rule classification.
  • [2026-06-03] Rethinking Video Generation Model for the Embodied World (ReVidGen / RBench / RoVid-X): RBench introduces the robot-oriented axis to the cluster — 25 models scored across 5 task domains × 4 embodiments with reproducible automatic sub-metrics (structural consistency, physical plausibility, action completeness). Achieves the strongest filed automatic-vs-human alignment (Spearman 0.96) of any cluster entry, and surfaces a sharp result that robotics-specific models trail general commercial video models on robotics evaluation.
  • [2026-05-24] Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation (PhyGenBench): PhyGenBench (ICML 2025) anchors the physical-commonsense axis — 160 T2V prompts spanning 27 physical laws across Mechanics / Optics / Thermal / Material Properties, with the three-stage PhyGenEval VLM-judge protocol (keyframe → multi-image order → full-video naturalness).
  • [2026-05-23] RISE-Video: Can Video Generators Decode Implicit World Rules?: RISE-Video introduces the implicit-world-rules axis for TI2V — 467 samples × 8 knowledge categories × 4 metrics (RA 0.4 / TC 0.25 / PR 0.25 / VQ 0.1) with a relaxed weighted score and a strict per-sample accuracy that exposes a 1.9–22.5% accuracy band hidden by 74.5–96.2% VQ.

Open questions

Section titled “Open questions”
  • Do these benchmarks correlate with each other on overlapping models? PhyGenBench, VideoPhy-2, and Physics-IQ all score Sora and Gen-3-class models — Physics-IQ ranks Sora last on physics but first on visual realism; PhyGenBench ranks Gen-3 first; VideoPhy-2 has Sora trailing Wan2.1-14B. No filed paper cross-validates rankings across the three.
  • Is the LMM-as-judge pipeline at the core of each benchmark itself validated against human raters at absolute accuracy (not just rank correlation)? RBench’s Spearman 0.96 is strong but methodology details for the auto-evaluator are pending arxiv fetch.
  • The “robotics-specific models trail general models” finding from RBench is striking — is this an evaluation artifact (general models look better on RBench’s chosen metrics) or a real capability gap? No filed paper triangulates Cosmos / DreamGen against Wan/Seedance on a non-RBench robotics task.
  • The “closed models don’t beat open” finding on VideoPhy-2 and Physics-IQ — does it survive when newer closed flagships (Sora 2, Veo 3, Kling 2.6) are run through the same protocol? Physion-Eval (Physion-Eval: Evaluating Physical Realism in Generated Video via Human Reasoning) tested those models but with a different metric.
  • How well do these benchmarks generalize to closed flagship products (Genie 3, Sora 2)? Genie 3 is absent from all filed leaderboards; only VideoPhy-2 (original Sora), Physics-IQ (original Sora), and RBench (Wan 2.6, Seedance 1.5 Pro) include closed-flagship entries.
  • Should there be a fourth cluster axis for temporal/motion benchmarks (Pulse of Motion, HumanScore, NewtonGen) — currently those sit under World Foundation Models and VLM-as-Evaluator but not under a dedicated benchmark concept?
  • Distilled 7B auto-evaluators (VideoPhy-2-Autoeval) beat much larger frozen VLM judges (Gemini-2.0-Flash-Exp). Is this a general pattern across the cluster, or specific to physics-commonsense?
  • Physics-IQ avoids VLM-judge weaknesses entirely by comparing pixel motion masks; PhyGenBench / VideoPhy-2 lean on VLM judges that themselves fail on physics. Is motion-mask comparison the right primitive when models eventually stop having static-camera mode (e.g., Sora-style autonomous cuts)?

Papers

Section titled “Papers”

2026-01

Section titled “2026-01”

2026-02

Section titled “2026-02”

2025-03

Section titled “2025-03”

2025-01

Section titled “2025-01”

2024-10

Section titled “2024-10”