Thinking with Video: Video Generation as a Promising Multimodal Reasoning Paradigm

TL;DR

Proposes “Thinking with Video” as the next reasoning substrate after “Thinking with Text” (CoT) and “Thinking with Images” (o3/o4-mini, Nano-Banana): the reasoning trace is a generated video, produced here by Sora-2. Introduces VideoThinkBench, covering vision-centric tasks (eyeballing puzzles, visual puzzles, ARC-AGI-2, mazes) and text-centric subsets (GSM8K, MATH-500, MMLU/MMLU-Pro, GPQA, AIME, MathVista, MathVision, MMMU, MMBench). On vision-centric tasks Sora-2 matches or beats GPT-5/Gemini-2.5-Pro/Claude-Sonnet-4.5; on text-centric tasks Sora-2 hits 98.9% GSM8K and 94% MATH-500 (audio-mode) but lags on AIME/GPQA. Self-consistency over time (Major Frame voting) and across retries lifts the verifiable-puzzle accuracy from 56% (last frame) to 90% (5-vote × major-frame).

Key claims

Sora-2 surpasses SOTA VLMs on eyeballing puzzles via “Major Frame” video voting: 40.2% average vs Claude 4.5 (35.1%), GPT-5 high (29.7%), Gemini 2.5 Pro (26.5%); Sora-2 hits 88% on Ray Intersection and 70% on Circle Center, where all three VLMs sit at 16–62% [Table 2].
On text-centric tasks, Sora-2’s audio output ≫ video output: 98.9% GSM8K, 92.0% MATH-500 (last frame 67.0%, audio 92.0%), 75.7% MathVista, 89.0% MMBench in the audio channel [Table 5]. Sora-2 struggles to render fully correct written solutions in the video frames [§2.3.3, §3.2.2].
On ARC-AGI-2 (1000-sample train set), Sora-2 scores 1.3% strict-match, on par with Gemini 2.5 Pro (1.9%) and GPT-5 high (0.5%), below Claude 4.5 (5.3%); manual inspection of 100 samples shows 3% Fully Correct + 14% Mostly Correct, with 42/55 failures being “Did Nothing” [Table 4, Table 10, §2.2.3].
Sora-2 is a few-shot learner — providing all ARC-AGI-2 demonstration examples yields more high-accuracy (0.65–1.0) outputs than 1-shot, confirming benefit from multiple examples [§3.1.1, Table 7].
Temporal self-consistency works like CoT self-consistency: Major-Frame voting beats Last-Frame (68% vs 56%); 5-try majority vote lifts Major-Frame to 90% on verifiable eyeballing puzzles. Generated videos sometimes end on SMPTE bars or black, so frame-averaging acts as a denoising filter [§3.1.2, Table 7].
Test-set leakage is unlikely to explain text-centric performance: when math problems are rewritten as new instances, Sora-2 retains comparable accuracy to the original test sets [§3.2.1]. The authors attribute much of the text-centric reasoning ability to Sora-2’s prompt rewriter upstream of the video model, not the video model itself [§3.2.3].

Method

VideoThinkBench has two halves. Vision-centric: 21 manually-designed eyeballing puzzles (Point/Line/Shape tasks, 50 samples each = 1050 total), 10 PuzzleVQA-derived color-filling and shape-drawing tasks, ARC-AGI-2, and mazes. Vision-centric instances are program-generated and verifiable (visual puzzles excepted). Text-centric: sampled subsets of GSM8K, MATH-500, AIME24/25, BBH, MMLU, MMLU-Pro, GPQA-diamond, SuperGPQA-easy, MathVista, MathVision, MMMU, MMBench, reformatted as (text prompt, reference image) pairs where the reference image renders the full problem on a white background.

Sora-2 is prompted to produce a video plus an audio track; for text-centric tasks the audio is required to be the final answer only, and the video carries the worked solution. Three evaluation channels are defined for Sora-2’s video output: Audio Evaluation (whisper-1 transcribes the audio, extract first phonetic alphabet token), Last Frame Evaluation (compute the average coordinate of red pixels on the last frame, pick the nearest option), and Major Frame Evaluation (sample every 5 frames, run the per-frame extractor, majority-vote across frames). LLM-as-Judge (GPT-4o) scores text-centric outputs against ground truth, with human alignment check.

Results

Vision-centric overall: Sora-2 average 41.7 vs Gemini 42.5 / GPT-5 46.2 / Claude 46.8 — within 5 points of frontier VLMs, despite being a video generator [Table 1].
Eyeballing puzzles (Major Frame): Sora-2 40.2 > Claude 35.1 > GPT-5 29.7 > Gemini 26.5 [Table 2].
Visual puzzles overall: Sora-2 66.2 vs Gemini 81.5 / GPT-5 86.8 / Claude 78.8; Sora-2 is competitive on color-filling (67.0 vs Gemini 73.9) but lags on shape-drawing (64.9 vs GPT-5 97.5) [Table 3].
Text-centric overall (audio): Sora-2 73.6 vs Gemini 83.7 / GPT-5 84.0 / Claude 81.2 [Table 5]. Sora-2 actually leads on MathVista (75.7 vs 67.5–72.5) and MMBench (89.0 vs 82.5–86.9).
Inference-time scaling: 5-vote × Major-Frame lifts Major-Frame option from 68% to 90% on verifiable puzzles [Table 7].
ARC-AGI-2: Sora-2 1.3%, Gemini 2.5 Pro 1.9%, GPT-5 high 0.5%, Claude 4.5 5.3% [Table 4].

Why it’s interesting

This is the natural next step in the “Thinking with Modalities” lineage Thinking with Modalities — explicitly framed as the post-Images successor to text-CoT and image-CoT, with the reasoning trace temporally extended and the model’s existing video generator doing all the work. It complements DiffThinker: Towards Generative Multimodal Reasoning with Diffusion Models (DiffThinker’s image-diffusion-as-CoT) and Demystifying Video Reasoning (which already showed pretrained video DiTs reason along the denoising trajectory) — Tong et al. operate one level up, treating Sora-2 as a black-box reasoner and measuring it against frontier VLMs on shared benchmarks. The finding that audio dominates video for text-centric answers (and the §3.2.3 attribution to Sora-2’s prompt rewriter) is a useful caution: a “thinking video model” benchmark may be silently scoring the upstream LLM rewriter, not the video model. The Major Frame × N-vote self-consistency boost (56→90%) is also a clean datapoint for inference-time-scaling on generation-side substrates, alongside the per-step iterative refinement axis in Self-Refining Video Sampling.