Do generative video models understand physical principles? (Physics-IQ)

TL;DR

Physics-IQ is a 396-video real-world benchmark (66 scenarios × 3 camera perspectives × 2 takes) for testing whether video generative models implicitly learn physical principles by predicting how a video continues. Models receive a 3-second conditioning clip (or its last frame for i2v models) and must generate the next 5 seconds; the prediction is scored against a held-out real-world continuation with four pixel-motion metrics (Spatial IoU, Spatiotemporal IoU, Weighted Spatial IoU, MSE) combined into a single Physics-IQ score normalized so the natural physical-variance ceiling sits at 100. Across Sora, Runway Gen 3, Pika 1.0, Lumiere, Stable Video Diffusion, and VideoPoet, the best model (VideoPoet multiframe) reaches only 24.1, and a Gemini-1.5-Pro 2AFC realism check shows Sora is by far the most visually convincing (55.6%, near chance) despite scoring last on physics — i.e., visual realism and physical understanding are essentially uncorrelated.

Key claims

Physics-IQ contains 396 real-world 8-second videos covering 66 physical scenarios across five categories — solid mechanics, fluid dynamics, optics, thermodynamics, magnetism — each filmed at 30 FPS from three camera perspectives with two takes per scenario to estimate physical variance [§Dataset].
The evaluation protocol splits each 8-second video into a 3-second conditioning clip and a 5-second ground-truth continuation; video2video (“multiframe”) models get up to 3 s of frames, image2video models get only the manually selected switch frame [§Evaluation protocol].
Physics-IQ is the first major video-physics benchmark built on real filmed footage rather than synthetic simulator output (e.g., Physion, IntPhys, PhyWorld), eliminating the real-vs-synthetic distribution shift that may confound results on models trained on natural videos [§Why create a real-world Physics-IQ dataset].
The Physics-IQ score combines four motion-based metrics — Spatial IoU (where action occurs), Spatiotemporal IoU (where + when), Weighted Spatial IoU (where + how much), MSE (how it looks) — and is normalized so the take-1-vs-take-2 physical-variance ceiling is 100 [§Metrics for physical understanding].
Across the eight evaluated model configurations, VideoPoet (multiframe) leads at 24.1, with Runway Gen 3 at 18.4, Lumiere (multiframe) at 18.2, Stable Video Diffusion at 13.5, Pika 1.0 at 9.5, and Sora last at 8.7 [Table 1].
Multiframe conditioning beats single-frame i2v at parity: both VideoPoet and Lumiere score higher in their multiframe variants than in their i2v variants [Table 1].
Visual realism, measured via a Gemini-1.5-Pro 2AFC test that asks the MLLM to pick the generated video out of a real/fake pair (chance = 50%), is decoupled from physical understanding: Sora achieves the best MLLM score (55.6%, near chance — most realistic) but ranks last on physics, while Lumiere multiframe is easily identified at 86.9% [Fig. 5].
The MLLM-realism vs Physics-IQ correlation across models is statistically insignificant, supporting the headline claim that visually realistic generation does not imply physical understanding [Fig. 5 right].
A causal model (VideoPoet) ranks first overall, suggesting causal next-frame prediction is at least as compatible with implicit physics as bidirectional diffusion baselines [§Physical understanding].
The benchmark and code are open-source under Apache 2.0 / CC-BY at github.com/google-deepmind/physics-IQ-benchmark; the dataset and protocol can be run by anyone with API access [§Dataset].

Method

Physics-IQ is constructed by manually staging 66 physically interesting scenarios (collisions, object continuity, occlusion, fluid mixing, chain reactions, gravity trajectories, material reactions, lights/shadows/reflections, magnetism) and filming each from three static camera positions (left, center, right) with two repeated takes using Sony Alpha a6400 cameras and 16-50mm lenses at 30 FPS. The two takes capture natural physical variance — even repeating the same drop or pour produces different exact trajectories — which sets the upper bound on what a model can possibly predict.

At evaluation time, each 8-second video is cut at a manually chosen “switch frame”: video2video models receive the 3-second preamble (downsampled to the model’s preferred FPS and resolution); image2video models receive only the switch frame. Models may also be given a human-written caption describing the scene without giving away the continuation. All evaluated models output a 5-second continuation, which is compared against the held-out 5-second ground truth.

The four motion metrics share a common substrate: a binary motion-mask video produced by thresholding pixel-intensity changes between consecutive frames (the static camera setup makes this clean). Spatial IoU collapses the masks across time with a max and computes IoU against the ground-truth motion map; Spatiotemporal IoU compares masks frame-by-frame; Weighted Spatial IoU averages per-frame action instead of taking the max, so a pendulum (repeated motion) and a one-shot rolling ball produce different signatures; MSE measures pixel-level appearance fidelity. The four are summed (MSE negated) and normalized by the physical-variance ceiling to produce the Physics-IQ score.

Visual realism is measured independently via a Gemini-1.5-Pro 2AFC psychophysics protocol: the MLLM is shown a randomized pair of one real and one generated video for the same scenario and must identify which is generated. Accuracy near 50% means the model has fooled the judge (realistic); accuracy near 100% means the judge always spots the fake.

Results

Best Physics-IQ score: VideoPoet multiframe 24.1 vs the physical-variance ceiling of 100.0 — a ~4× gap on a score where the upper bound is just “replay the same physical phenomenon a second time” [Table 1].
Full leaderboard: VideoPoet multiframe 24.1 > Runway Gen 3 18.4 > Lumiere multiframe 18.2 > VideoPoet i2v 18.0 > Lumiere i2v 17.1 > Stable Video Diffusion 13.5 > Pika 1.0 9.5 > Sora 8.7 [Table 1].
Multiframe ≫ i2v on the same backbone: VideoPoet 24.1 vs 18.0; Lumiere 18.2 vs 17.1 [Table 1].
Even the weakest absolute metric (Spatial IoU — just “where did action happen”) shows a large model-vs-real gap (best 0.245 vs physical-variance 0.645), so the gap is not an artifact of overly strict metrics [§Physical understanding, Table 1].
Per-category breakdown (Fig. 6) shows performance varies but no category is “solved”; certain Solid Mechanics and Magnetism scenarios are essentially zero for all models.
Sora wins visual realism with 55.6% MLLM 2AFC accuracy (chance = 50%, lower = more realistic), Runway Gen 3 second at 74.8%, VideoPoet multiframe third at 77.3%, Lumiere multiframe worst at 86.9% [§Visual realism, Fig. 5 left].
Sora is simultaneously most realistic and worst on physics; the cross-model correlation between MLLM-realism and Physics-IQ is statistically insignificant [Fig. 5 right].
Qualitatively, Sora is penalized by frequent unprompted transition cuts (despite being told to keep a static camera); SVD over-generates physically implausible motion, hurting Spatial/Spatiotemporal IoU [§Physical understanding].

Why it’s interesting

Physics-IQ is the real-world anchor of the wiki’s video-physics-benchmark cluster — Video Generation Benchmarks currently catalogs PhyGenBench, VideoPhy-2, RISE-Video, RBench, and Dream.exe, all of which are either prompt-based (text→video commonsense) or simulator-based, whereas Physics-IQ films staged physical events with two takes per scenario and uses the take-1-vs-take-2 difference as the natural ceiling. That makes it a complementary axis to Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation (PhyGenBench) (PhyGenBench’s prompt-based VLM-judge protocol) and VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generation (action-centric human-rated PC/SA): instead of asking a VLM judge whether a video looks plausible, Physics-IQ compares generated motion masks against actual ground-truth motion masks. The “Sora most realistic, worst on physics” finding is the clearest filed evidence that visual realism and physical understanding decouple, and it directly motivates the wiki’s Inference-time Physics Alignment of Video Generative Models with Latent World Models line, which uses VJEPA-2 surprise as a Physics-IQ-style physics reward and reports MAGI-1 24B improving from 55.22 → 62.00 on Physics-IQ via best-of-N rollout (one of the few model configurations on the leaderboard that meaningfully closes the gap). For Luma: the dataset, code, and metrics are open, the protocol matches the way internal i2v and v2v models are trained, and the small 66-scenario size makes it cheap to run as a physics-targeted eval gate alongside FVD/LPIPS.