Apple-π: Benchmarking Thinking with Video Towards Law-Grounded Physical Intelligence

TL;DR

Apple-π is a benchmark that evaluates whether video generation models actually reason with physical laws or merely produce motion that looks plausible. It comprises three parts: Orchard, a dataset of 400 videos spanning 10 classical-mechanics tasks (split into single-law and multi-law subsets); a three-stage protocol — Perception → Formulation → Deduction — that treats the generated video as the model’s visible reasoning trace via chain-of-frames prompting on infographic-annotated first frames; and a hybrid evaluation suite that combines MLLM-based subjective scoring with physics-law-grounded objective measures. Across 11 evaluated models, the best video model scores only 0.473, exposing a Perception→Formulation→Deduction bottleneck, weak multi-law state transfer, and a persistent Sim-to-Real gap.

Key claims

Current video models are far from reliable law-grounded world simulators: best video model scores only 0.473 on the full three-stage protocol across 11 evaluated models [Abstract].
The failure profile is stage-structured: models perceive physics but struggle to formulate laws and generate law-consistent dynamics, forming a Perception → Formulation → Deduction bottleneck [Abstract].
Multi-law state transfer is a distinct failure axis from single-law tasks: the multi-law subset probes generalization while single-law tasks provide confounder-free diagnosis [Abstract].
A persistent Sim-to-Real gap shows up under source-resolved analysis, orthogonal to per-stage scores [Abstract].
Chain-of-frames prompting on infographic-annotated first frames operationalizes “thinking with video” as an evaluable protocol — the generated video is treated as the model’s visible reasoning trace [Abstract].
Hybrid MLLM subjective + physics-law-grounded objective evaluation is the design contract of the evaluation suite, enabling stage-resolved diagnosis of where a model fails, not only whether it fails [Abstract].

Method

Apple-π is structured as a three-component benchmark. Orchard curates 400 videos covering 10 canonical classical-mechanics tasks, with single-law tasks isolated from multi-law tasks so that generalization probes are separable from confounder-free per-law diagnosis. The Benchmark Protocol decomposes scientific reasoning into three stages executed via chain-of-frames prompting: given an infographic-annotated first frame that specifies the initial state (masses, velocities, forces, geometry), the model must (i) perceive the scene, (ii) formulate the governing law, and (iii) deduct the resulting dynamics by generating the continuation video — which the benchmark treats as the model’s reasoning trace. The Evaluation Suite is hybrid: an MLLM judge scores subjective aspects, and a physics-law-grounded objective evaluator scores dynamics against the ground-truth law. The two are combined into stage-, pillar-, and source-resolved diagnostic scores.

Results

Best of 11 evaluated video models: 0.473 overall [Abstract] — no model approaches reliable law-grounded simulation.
Stage decomposition surfaces a Perception → Formulation → Deduction bottleneck: models can perceive the physical setup but fail to formulate laws and generate law-consistent dynamics [Abstract].
Multi-law tasks systematically degrade performance relative to single-law tasks, indicating weak physical-state transfer across composed laws [Abstract].
A Sim-to-Real gap persists under source-resolved analysis: models trained/evaluated on simulator content do not transfer their apparent physical fluency to real-world sources [Abstract].

Why it’s interesting

Apple-π sits at the intersection of two active wiki threads. First, it extends the Video Generation Benchmarks cluster’s central move — replacing surface-visual-quality scores with capability-specific probes — by anchoring evaluation explicitly in physical laws rather than in physical commonsense (VideoPhy-2, Physics-IQ) or implicit world rules (RISE-Video). Where Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation (PhyGenBench) organizes prompts by 27 physical laws across 4 domains and VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generation organizes by 197 real-world actions, Apple-π adds a third axis: 10 canonical mechanics tasks × law-grounded objective measures, with an explicit stage decomposition (Perception → Formulation → Deduction) that makes failure localizable. The 0.473 ceiling is directly comparable to VideoPhy-2’s 32.6% joint SA∧PC and RISE-Video’s 22.5% strict-accuracy ceiling — all three benchmarks converge on the same finding that visual plausibility is largely solved and physical reasoning is largely open.

Second, it operationalizes the Thinking with Modalities cluster’s substrate-(b) diffusion-native reasoning position — video generation as the reasoning trace — as an evaluation methodology. This is the natural dual to Demystifying Video Reasoning‘s finding that pretrained video DiTs already exhibit Chain-of-Steps reasoning: if the reasoning trace lives in the diffusion trajectory, then chain-of-frames prompting on infographic-annotated first frames is the right probe for it. Apple-π complements Thinking with Video: Video Generation as a Promising Multimodal Reasoning Paradigm by supplying a diagnostic protocol tailored to the “video generation as reasoning paradigm” position. For the World Foundation Models cluster, it adds a benchmark-as-probe entry alongside RISE-Video: Can Video Generators Decode Implicit World Rules? and Do generative video models understand physical principles? (Physics-IQ) with the sharpest per-stage decomposition on file.