A Very Big Video Reasoning Suite (VBVR)

TL;DR

VBVR is a video-reasoning evaluation suite built on the bet that video reasoning is “the next fundamental intelligence paradigm, after language reasoning” — a benchmark + data-factory project that ships task-specific generators for spatiotemporal embodied reasoning problems (domino chain reactions with branching paths, decreasing-size sequence completion, 10×10 grid shortest-path, object rotate-and-align, color-pair connection, mirror reflection). Each task is paired with a programmatic data generator (GitHub-released, one repo per task family) that produces (first-frame, prompt, last-frame, ground-truth video) tuples. The project page exposes a leaderboard comparing ten video generators — VBVR-Wan2.2, CogVideoX 1.5, Kling 2.6, LTX-2, Runway Gen-4, Sora 2, Veo 3, Wan 2.2 I2V, Hunyuan I2V, Seedance 2.0 — and the team’s own VBVR-Wan2.2 fine-tune. The framing is that current video generators look plausible per-frame but fail at deterministic visual-reasoning tasks with checkable ground truth.

Key claims

Video reasoning is positioned as the successor paradigm to language reasoning, on the argument that spatiotemporal embodied world experience is more naturally captured in video than in text [home page introduction].
Each task family in the suite has its own programmatic data generator released as a GitHub repo under the VBVR-DataFactory org — six tasks visible on the landing page (O-23_domino_chain_branch_path_prediction, G-133_select_next_figure_decreasing_size_sequence, G-18_grid_shortest_path, G-25_seperate_object_spinning, G-54_connecting_color, plus a Mirror Reflection family) with task IDs hinting at a much larger taxonomy [GitHub links in body].
The evaluation format is image-to-video with first-frame + last-frame ground truth: prompts describe a deterministic visual process (e.g. “dominos fall step by step from left to right, turning darker as they fall … the gray trunk falls first, then splits into the red Branch A (upward) and cyan Branch B (downward)”) and the generated video is checked against ground-truth first/last frames plus the intermediate process [task descriptions].
The benchmark covers ten contemporary video generators plus the team’s own VBVR-Wan2.2 fine-tune — CogVideoX 1.5, Kling 2.6, LTX-2, Runway Gen-4, Sora 2, Veo 3, Wan 2.2 I2V, Hunyuan I2V, Seedance 2.0 — making it directly comparable to RISE-Video’s 11-model TI2V leaderboard [model list in samples section].
The leaderboard is structured along (Modality, Split, Type, Category) facets, implying per-category breakdowns rather than a single scalar [leaderboard section].

Method

VBVR is best understood as a data-factory + benchmark pair. The “data factory” half is a collection of programmatic task generators — each one a Python repo that procedurally synthesizes (input image, text prompt, ground-truth video) tuples for a single visual-reasoning task family. Tasks visible on the landing page span chain-reaction physics (domino falls with branching paths), abstract pattern continuation (selecting the next figure in a decreasing-size sequence), grid-world planning (10×10 shortest-path with a moving agent), geometric manipulation (rotate-and-translate objects to match dashed targets), constraint-satisfaction layout (connecting same-color shapes without crossings), and mirror reflection. Task IDs prefixed O- and G- (with numbers up to at least 133) suggest a much larger underlying taxonomy than the six samples shown.

The “benchmark” half exposes a leaderboard that scores ten video generators plus the team’s own VBVR-Wan2.2 fine-tune on each task. The leaderboard UI exposes filters for Modality / Split / Type / Category, suggesting the suite supports per-axis stratification. The landing page shows representative model outputs side-by-side with the ground-truth first/final frames; the evaluation protocol itself (how generated videos are scored against ground truth, whether by VLM-as-judge, frame-level comparison, or task-specific symbolic checkers) is not detailed in the public page text.

Results

The landing page does not surface concrete leaderboard numbers in the fetched body, but the structural facts are clear: ten production video generators are evaluated on the same task suite, the team’s VBVR-Wan2.2 fine-tune sits at the top of the displayed sample carousel for each task, and per-task qualitative outputs are shown for tasks like Mirror Reflection. The methodological move that matters is that ground-truth videos exist (because the generators are programmatic), so unlike RISE-Video’s LMM-as-judge protocol the scoring can in principle be reduced to deterministic frame-level checks against the procedurally-generated reference.

The implicit headline result — consistent with the rest of the video-reasoning literature filed to date — is that frontier video generators (Sora 2, Veo 3, Kling 2.6, Runway Gen-4) produce plausible-looking outputs but fail the deterministic visual-reasoning check on most tasks, leaving a wide gap that a task-specific fine-tune (VBVR-Wan2.2) can close.

Why it’s interesting

VBVR is the second video-reasoning benchmark filed in the same week as RISE-Video and takes a structurally different bet: RISE-Video uses 467 human-annotated samples + LMM-as-judge over human-written per-sample questions, whereas VBVR uses procedurally-generated tasks with deterministic ground-truth videos. The two approaches answer different questions — RISE-Video probes broad knowledge categories (commonsense, societal, experiential, temporal) where reasonable answers vary and an LMM-judge is the only practical scorer; VBVR probes a narrower set of deterministic visual-reasoning tasks where ground-truth videos are available by construction and scoring can in principle be exact. Both report a large gap between per-frame visual quality and reasoning correctness on essentially the same model set (Sora 2, Veo 3, Kling, Wan, CogVideoX, LTX, Hunyuan, Seedance), which is a useful triangulation on the same underlying claim.

The data-factory framing is the part that’s new: rather than ship a static benchmark, VBVR ships generators (one GitHub repo per task family) — so the benchmark can be grown indefinitely and contamination is structurally avoided. This pairs naturally with the World Foundation Models cluster’s running question of whether video generators decode implicit world rules: VBVR turns the question into a procedural test bed where the rules are by construction known and checkable. For Luma, this is a candidate eval to run Luma’s own video models against, with the caveat that the public leaderboard already includes a VBVR-tuned Wan baseline (VBVR-Wan2.2) — i.e. the suite has already been used as a fine-tuning target by the authors, so headline numbers should be read with that contamination in mind.