Video models are zero-shot learners and reasoners

TL;DR

A Google DeepMind position paper arguing that large generative video models are entering their “GPT-3 moment” for machine vision. The authors prompt Veo 3 (and the older Veo 2 as a scaling baseline) across 18,384 generated videos spanning 62 qualitative and 7 quantitative tasks it was never trained for — edge detection, segmentation, super-resolution, intuitive physics, image editing, maze solving, visual analogy, visual symmetry — and find that Veo 3 succeeds zero-shot on a wide span. They name the underlying primitive “chain-of-frames” (CoF) reasoning: the video model applies stepwise changes across time and space the way a language model applies stepwise changes across tokens. The Veo 2 → Veo 3 jump is large and consistent (e.g. 5×5 maze pass@10 14% → 78%), so the authors argue trend lines, not absolute numbers, are what matters.

Key claims

The minimal recipe is “prompt Veo via the Vertex AI API, 8s at 720p/24 fps, no fine-tuning, no task-specific heads” — mirroring the NLP shift from task-specific fine-tuning to prompting a generalist model [§2].
Veo 3 performs four hierarchical visual capabilities zero-shot: perception (edge detection, segmentation, keypoint, super-resolution, deblurring, denoising), modeling (intuitive physics — flammability, rigid/soft body dynamics, buoyancy, Visual Jenga ordering), manipulation (background removal, style transfer, colorization, inpainting/outpainting, novel-view generation), and reasoning (graph traversal, visual BFS, Sudoku, mazes, navigation, rule extrapolation) [§3, Figs. 10–59].
Quantitative reasoning headline: 5×5 maze pass@10 jumps from 14% (Veo 2) to 78% (Veo 3); on 9×9 mazes Veo 3 beats Gemini 2.5 Pro reading the maze as an image (I2T), though Gemini wins on small mazes given an ASCII representation (T2T) [§4.5, Fig. 7].
Quantitative perception headline: Veo 3 hits 0.77 pass@10 on edge detection (OIS) vs 0.90 for task-specific SOTA; achieves mIoU 0.74 on instance segmentation, on par with Nano Banana (0.73) but below SAMv2 [§4.1, §4.2, Figs. 3–4].
LLM-only sanity checks rule out the Vertex prompt-rewriter as the secret reasoner: Gemini 2.5 Pro from the input image alone cannot reliably solve robot navigation, mazes, or visual symmetry, isolating the win to the video generation step [§2].
pass@10 consistently beats pass@1 across all quantitative tasks with no plateau, naming inference-time scaling (sampling more candidates) as the obvious next compute lever — analogous to how LLMs benefited from temperature/sampling-based test-time compute before RLHF and verifiers were added [§5, Discussion].
“Performance is a lower bound”: for any given task, the report depends on a particular visual + textual prompt; the authors flag prompt engineering as essential and document a 40–64 percentage-point swing across prompts on visual symmetry alone [§5, App. C].
Documented failure modes: monocular depth, surface normal estimation, “fold this laundry” visual instruction following, planning to move a sofa through a small door, reflect/rotate visual analogies (systematic bias below chance 0.33), certain visual puzzles [§4.7 Fig. 9, App. D Figs. 70, 76, 77].
Cost framing: video generation is currently more expensive than a task-specific model, but the authors cite Epoch AI’s 9–900× per-year inference cost decline for LLMs at fixed quality as the analog they expect for video [§5 “Video generation is expensive…”].

Method

The paper has no new model and no fine-tuning. Veo 2 (December 2024) and Veo 3 (May 2025) are queried as black boxes via Google Cloud’s Vertex AI API; the only inputs are an initial image (used as the first frame) and a text instruction. Outputs are 8s, 24 fps, 720p, 16:9 videos. The Vertex API uses an LLM-based prompt rewriter, which the authors treat as part of the system; for the high-stakes claims (mazes, symmetry, robot navigation) they verify that Gemini 2.5 Pro from the input image alone cannot solve the task. Each quantitative task is scored under both “best frame” (performance ceiling across any frame) and “last frame” (predetermined, deployable) protocols. Reasoning tasks are evaluated with automatic verifiers: maze paths are checked for legality and reaching the goal; symmetry is checked cell-by-cell against the ground-truth reflection; analogies are scored against KiVA’s transformation labels [§B.5–B.7].

Results

Maze solving (5×5): Veo 2 14% pass@10 → Veo 3 78%. Nano Banana matches Veo 3 on rectangular mazes but fails entirely on irregular ones; Gemini 2.5 Pro wins at small T2T mazes but is worse at I2T or 9×9 [§4.5, Fig. 7].
Edge detection: Veo 3 0.77 OIS pass@10 vs 0.90 task-specific SOTA; some Veo edge maps are more detailed than the human-annotated ground truth [§4.1, Fig. 3].
Segmentation: Veo 3 mIoU 0.74 best-frame pass@10, matching Nano Banana (0.73); below SAMv2 but with the green-background prompt outperforming white-background by 8 points [§4.2, Fig. 4].
Object extraction (counting via animal lineup): Veo 3 reaches 93% pass@10; Veo 2 is near chance [§4.3, Fig. 5].
Visual symmetry solving: Veo 3 outperforms Veo 2 and Nano Banana by a large margin; prompt choice swings pass@1 by 40 pp (shape split) and 64 pp (random split) [§4.6, Fig. 8, App. C].
Visual analogy: Veo 3 solves color and resize transformations above chance, but both Veo 2 and Veo 3 perform below the 0.33 chance baseline on reflect and rotate — a clean systematic-bias failure [§4.7, Fig. 9].
Trendline: Veo 2 → Veo 3 step-change in performance across nearly every task, released ~6 months apart, with no signs of saturation in pass@1 → pass@10 [§5].

Why it’s interesting

This is the paper the wiki’s Thinking with Modalities concept page already treats as foundational — the substrate (b) “diffusion-native parallel exploration” claim ultimately traces back to Veo 3’s emergent maze-solving, and the chain-of-frames terminology this paper introduces is the direct progenitor of the chain-of-steps (CoS) substrate analyzed by Demystifying Video Reasoning and the early-plan-commitment dynamics studied by Video Models Reason Early (arXiv 2603.30043, cited in Demystifying Video Reasoning §References). It also anchors the generative-rollout pole of World Foundation Models alongside Project Genie: Experimenting with infinite, interactive worlds — both Google DeepMind, both flagship closed video models — and provides the empirical scaffold for the normative claim in The flavor of the bitter lesson for computer vision that video generative pre-training is the right perception-side objective for embodied AI. The pass@1 → pass@10 inference-time scaling result connects directly to Inference-Time Scaling.

The result is also a useful corrective on the AGI Is Not Multimodal line of argument: a single large generative video model does appear to span perception → modeling → manipulation → reasoning under a unified objective, exactly what “AGI is not multimodal” predicts shouldn’t be the path. The failure modes — reflect/rotate below chance, depth/normals broken — are also informative: they suggest the model has learned ego-centric / appearance-based priors but not the abstract symmetry group structure, which is consistent with the systematic-bias observations on rotation analogies in other vision benchmarks.