Skip to content
Bud's Wiki

Video Generation Models are General-Purpose Vision Learners

TL;DR

Section titled “TL;DR”

GenCeption (Google DeepMind, ECCV 2026) takes a pre-trained text-to-video diffusion backbone and turns it into a single feed-forward, text-instruction-steered vision model that outputs depth, surface normals, camera pose, expression-referring segmentation, and 3D keypoints — matching or beating task-specific SOTA (DepthAnything3, SAM3, D4RT, VGGT-Ω, Sapiens, Lotus-2) with 7×–500× less training data. The framing is the video analog of the “generation-is-pretraining” argument: what next-token prediction was to language and image generation was to 2D vision (Vision Banana), text-to-video generation is to spatiotemporal visual intelligence. The paper also reports that a model fine-tuned only on synthetic human videos transfers zero-shot to real footage and to out-of-distribution categories (animals, robots).

Key claims

Section titled “Key claims”
  • Large-scale text-to-video generation provides the necessary spatiotemporal priors, vision–language alignment, and scalability to serve as a general-purpose pre-training paradigm for computer vision [Abstract].
  • A pre-trained video diffusion backbone can be converted into a single feed-forward perception model steered by text — not multi-step generation — while retaining the backbone’s priors [Abstract].
  • GenCeption matches or surpasses task-specific SOTA across depth, surface normals, camera pose, expression-referring segmentation, and 3D keypoint prediction — competing directly against DepthAnything3, SAM3, D4RT, VGGT-Ω, Sapiens, David, Genmo, and Lotus-2 with one model [Abstract].
  • Video-generation pretraining outperforms alternative predictive-latent video pretraining (V-JEPA, VideoMAE) under comparable settings [Abstract].
  • GenCeption achieves comparable performance to leading models like D4RT and VGGT-Ω with 7–500× less training data, indicating exceptional data efficiency [Abstract].
  • Preliminary data- and model-scaling trends are reported for the video-gen-as-pretraining recipe [Abstract].
  • A model trained exclusively on synthetic human videos generalizes to real-world footage and out-of-distribution object categories (animals, robots) [Abstract].

Method

Section titled “Method”

GenCeption starts from a pre-trained text-to-video generative diffusion backbone and repurposes it as a feed-forward perception model: instead of running the full iterative denoising loop, the model produces per-task predictions in a single forward pass, with the task selected by a text instruction. The published abstract asserts that one set of weights covers dense outputs (depth, surface normal maps), geometric quantities (camera pose), sparse structured outputs (3D keypoints), and language-grounded segmentation (expression-referring). The mechanism by which sparse and language-conditioned outputs are re-parameterized against a video-diffusion backbone’s native output space is not disclosed in the abstract; the full paper is needed to specify the task-head design and how instruction conditioning routes between dense-image and sparse outputs.

The comparison against V-JEPA / VideoMAE (also video pretraining, but predictive-latent rather than generative) is run “under comparable settings” per the abstract, positioning generative video pretraining as strictly stronger than latent-predictive video pretraining for downstream perception at matched compute — a stronger claim than Vision Banana made on the image side.

Results

Section titled “Results”
  • Matches or surpasses task-specific SOTA across the reported task suite (depth, normals, camera pose, expression-referring segmentation, 3D keypoints) against DepthAnything3, SAM3, D4RT, VGGT-Ω, Sapiens, David, Genmo, and Lotus-2 [Abstract].
  • Beats V-JEPA and VideoMAE video pretraining under comparable settings [Abstract].
  • Comparable to D4RT and VGGT-Ω on their target tasks with 7×–500× less training data [Abstract].
  • Zero-shot generalization from synthetic-human-only fine-tuning to real footage and to animals / robots is reported qualitatively [Abstract; project page].

The abstract does not provide per-benchmark numerical tables, ablations on backbone choice or on data-mixture ratios, or scaling curves. Empirical claims here are all reproduced from the paper’s abstract; per-task quantitative verification requires the full PDF.

Why it’s interesting

Section titled “Why it’s interesting”

This is the video-side companion to Image Generators are Generalist Vision Learners (Vision Banana) (Vision Banana), and the two together stake out a strong “generation-is-vision-pretraining” position from Google DeepMind — the image paper claimed generation pretraining beats SAM/Depth-Anything specialists on 2D tasks; this paper claims a video generative backbone beats DepthAnything3, SAM3, VGGT-Ω, Sapiens, and Lotus-2 while also covering inherently spatiotemporal tasks (camera pose, 4D keypoints) that a still-image generator cannot. It reads as the empirical version of the The flavor of the bitter lesson for computer vision thesis: explicit 3D reconstruction pipelines and per-task pretext objectives (V-JEPA / VideoMAE) are hand-crafted intermediates that video-generative pretraining dissolves. The V-JEPA-comparison line is the direct empirical response to LeCun’s Yann LeCun: World Models — Enabling the next AI revolution (ETH Frontiers of Embodied AI) latent-predictive framing — and unlike the image-side Vision Banana, this paper does run that head-to-head comparison. The synthetic-humans-to-real-world generalization also updates Pose Estimation and Motion Capture: it complements ProxyPose: 6-DoF Pose Tracking via Video-to-Video Translation‘s “video diffusion transfers zero-shot across sensing modalities” claim by adding zero-shot transfer across object categories on the same backbone family, and complements Video models are zero-shot learners and reasoners‘s Veo-3-as-zero-shot-reasoner argument by pushing the same recipe from prompt-only inference to a light task-tune.

See also

Section titled “See also”