A Very Big Video Reasoning Suite

TL;DR

VBVR is a video-reasoning suite with three components: (1) VBVR-Dataset — a procedurally-generated training corpus of 1,000,000 training samples + 7,500 test samples across 200 tasks, ≈1000× larger than prior video-reasoning datasets; (2) VBVR-Bench — a rule-based, non-LMM evaluation toolkit with task-specific scorers and a dual in-domain / out-of-domain split; (3) VBVR-Wan2.2 — a Wan2.2-I2V-A14B fine-tuned on VBVR-Dataset that posts a new SOTA of 0.685 overall (84.6% relative gain over base) and exposes early evidence of scaling-driven OOD generalization. Tasks are organized into five cognitive faculties grounded in Aristotelian/Kantian taxonomy (Abstraction, Knowledge, Perception, Spatiality, Transformation), each implemented as a parameterized generator that emits (first_frame, prompt, final_frame, ground_truth.mp4). The headline empirical claim is that frontier video generators (Sora 2 at 0.546, Veo 3.1 at 0.480) still trail humans (0.974) by a wide margin on deterministic visual reasoning, and that ID/OOD performance both plateau with data scale — a persistent gap not closed by more data alone.

Key claims

VBVR-Dataset contains 2,015,000 images and 1,007,500 video clips across 200 tasks (1,000,000 train across 100 train tasks at 10,000 per-task; 7,500 test across 150 test tasks at 50 per-task) — roughly three orders of magnitude larger than the next-largest prior video-reasoning dataset (VR-Bench at 7,920 videos, MMGR at 530) [§3.2, Tab. 1].
Task taxonomy is grounded in five cognitive faculties — Abstraction, Knowledge, Perception, Spatiality, Transformation — with explicit philosophical and neuroscientific lineage (Aristotle’s dunameis, Kant’s Einbildungskraft, hippocampal place/grid cells, prefrontal modularity) [§3.1].
Each task is implemented as a parameterized generator emitting a four-tuple (first_frame.png, prompt.txt, final_frame.png, ground_truth.mp4) — generators undergo design review (>200 approved from >500 proposals) against six quality criteria including deterministic solvability and video dependency [§3.3.1, §3.3.2].
VBVR-Bench explicitly rejects VLM-as-judge in favor of rule-based, task-specific scorers with sub-criteria like spatial accuracy, trajectory correctness, temporal consistency, logical validity — defended on reproducibility, determinism, granular verifiability, and freedom from LLM hallucination [§4.1].
Human-preference alignment is validated by a large-scale pairwise-preference study; per-model win ratios computed from VBVR-Bench’s automatic scorers correlate strongly positively with win ratios from human pairwise annotations [§4.2, Fig. 4].
Leaderboard cluster: open-source models 0.27–0.37 overall (Wan2.2-I2V-A14B best at 0.371); proprietary models led by Sora 2 (0.546) and Veo 3.1 (0.480), Kling 2.6 (0.369) and Runway Gen-4 Turbo (0.403) trailing; the strongest model still falls far short of human (0.974) [§4.3, Tab. 3].
VBVR-Wan2.2 (Wan2.2 fine-tuned on VBVR-Dataset with no architectural changes) reaches 0.685 overall, an 84.6% relative gain over its base Wan2.2 (0.371) and the new SOTA — strongest in Spatiality (0.745) and Transformation (0.833) on the ID split [§4.3, Tab. 3].
Scaling experiments under fixed architecture show concurrent ID and OOD gains, with both plateauing as data scale grows — and a persistent ID↔OOD gap (0.760 vs 0.610 for VBVR-Wan2.2) that does not close with more data [§5.2, Tab. 3].
Residual capability-correlation analysis (general-factor regressed out) reveals non-trivial structure: Knowledge ↔ Spatiality strongly positive, Knowledge ↔ Perception strongly negative, Abstraction ↔ Transformation strongly negative — interpreted via known cortical/hippocampal modularity [§4.4, Fig. 5].
150 of 200 tasks are publicly released; 50 are held out as a hidden test set to preserve leaderboard integrity from contamination [§3.2].

Method

VBVR is a data-factory + benchmark + reference-model package. The data factory side organizes contributor-submitted task designs around five cognitive faculties (Abstraction, Knowledge, Perception, Spatiality, Transformation), each grounded in cited philosophical and neuroscientific literature. Approval gates require information sufficiency (all reasoning cues in first-frame + prompt), deterministic solvability (unique verifiable success criterion), video dependency (not solvable from a static image), visual clarity, parametric diversity (≥10,000 non-trivial instances per task), and technical feasibility. ~200 of >500 submitted proposals cleared this gate. Each approved task is then implemented as a Python generator that procedurally synthesizes (first_frame.png, prompt.txt, final_frame.png, ground_truth.mp4) tuples — components (1)+(2) are model inputs, components (3)+(4) provide verifiable supervision including the complete solution trajectory.

The benchmark side (VBVR-Bench) is structured as a dual split: 50 in-domain test tasks share categories with training but use unseen parameter configurations; 50 out-of-domain tasks are entirely novel categories never seen at training time. Each task carries a dedicated rule-based scorer — for the worked example (Task G-45: Key Door Matching), the scorer decomposes into target-identification accuracy (30%), path validity (30%), path efficiency vs BFS-optimal (20%), and animation quality (20%). This deliberately substitutes for VLM-as-judge: the paper argues that the existence of unique, verifiable correct answers makes rule-based scoring feasible and more reproducible than LLM-based judgments, and validates the rule-based scorers against human preference annotations (Spearman correlation reported in §4.2, with strong agreement; exact numeric clipped in the fetch).

The reference-model side fine-tunes Wan2.2-I2V-A14B on the VBVR training split with no architectural modifications, yielding VBVR-Wan2.2. The scaling study traces both ID and OOD performance against training data scale on this fixed architecture to isolate data-scaling effects from architectural confounds.

Results

Headline leaderboard (overall score on the 100-task test split, weighted across ID and OOD): humans at 0.974, VBVR-Wan2.2 at 0.685, Sora 2 at 0.546, Veo 3.1 at 0.480, Runway Gen-4 Turbo at 0.403, Wan2.2-I2V-A14B (base) at 0.371, Kling 2.6 at 0.369, LTX-2 at 0.313, CogVideoX1.5-5B-I2V and HunyuanVideo-I2V tied at 0.273 [§4.3, Tab. 3]. ID/OOD split: VBVR-Wan2.2 scores 0.760 ID / 0.610 OOD; Sora 2 scores 0.569 ID / 0.523 OOD; the SOTA (VBVR-Wan2.2) carries a 0.150 ID↔OOD gap — visible across the entire leaderboard, indicating that even with 1M training samples the suite tests a generalization axis that data scaling alone does not close [§4.3, Tab. 3]. VBVR-Wan2.2 wins all five faculty categories on ID and four of five on OOD (Abstraction OOD is led by Sora 2 at 0.546, with VBVR-Wan2.2 close behind at 0.768 — note Sora 2’s Abstraction OOD figure is 0.546, the table column header for OOD reads differently; the relevant fact is that VBVR-Wan2.2 is best across the board) [Tab. 3].

Scaling behavior under fixed Wan2.2 architecture: ID and OOD both improve concurrently with training-data scale, with qualitative reports of emergent instruction-following, controlled editing, and semantic understanding at larger scale [§5.2, §5.3]. Both curves plateau before reaching human performance, and the ID↔OOD gap persists — read by the authors as evidence of architectural bottleneck, not data starvation [§5.2].

Capability-correlation analysis (with general-factor regressed out): Knowledge–Spatiality strongly positive (linked by the authors to hippocampal cognitive-map literature); Knowledge–Perception strongly negative (read as core-knowledge being perceptual rather than propositional); Abstraction–Transformation strongly negative; most other pairs near zero [§4.4, Fig. 5]. The point of this section is less the individual coefficients than the claim that VBVR-Bench supports interpretable diagnosis of how reasoning capabilities co-develop, not just a scalar ranking.

Why it’s interesting

VBVR is the paper version of the A Very Big Video Reasoning Suite (VBVR) project page already filed (which exposed the leaderboard and task generators) — but the paper adds the parts the landing page didn’t: scale (200 tasks, 1M training samples, ≈1000× prior datasets), the rule-based scoring rationale, human-preference alignment validation of those scorers, and the scaling-curve evidence that data alone plateaus before humans. The most consequential design choice — and the part worth highlighting for the team — is the explicit rejection of VLM-as-judge in favor of task-specific rule-based scorers. This puts VBVR in direct methodological tension with RISE-Video: Can Video Generators Decode Implicit World Rules? (RISE-Video, same week), which uses LMM-as-judge against human-written per-sample questions for the same kind of video-reasoning probe. Both benchmark essentially the same model set (Sora 2, Veo 3.1, Kling 2.6, Wan 2.2, CogVideoX, LTX, Hunyuan, Runway Gen-4) and both report that frontier generators fall short of humans — but they answer different questions about how video reasoning should be scored.

The scaling result is the part with the longest half-life: under fixed Wan2.2 architecture and 1M training samples, ID and OOD both plateau, and the ID↔OOD gap persists. Read together with The flavor of the bitter lesson for computer vision‘s argument that generative-rollout WFMs should be the perception-side pre-training substrate, VBVR adds an empirical caveat: at least within the I2V architecture class, more data is not the missing ingredient for video reasoning. This complements On the Slow Death of Scaling‘s broader thesis about scaling plateaus, applied here to a specific capability axis. For Luma, VBVR-Wan2.2’s 84.6% relative gain over Wan2.2 baseline is also a useful datapoint that structured synthetic reasoning data is a high-leverage post-training signal — connecting to the Synthetic Training Data thesis the wiki has been building.