DVD: Deterministic Video Depth Estimation with Generative Priors

TL;DR

DVD adapts a pre-trained video diffusion model (Wan 2.1) into a deterministic, single-pass video depth estimator — bypassing both the stochastic geometric hallucinations of generative depth predictors (Marigold-style) and the data hunger of discriminative ones (Depth Anything V2 / Video Depth Anything). Three design choices make the adaptation work: repurposing the diffusion timestep as a “structural anchor”, a latent manifold rectification (LMR) loss that prevents regression-induced over-smoothing, and a global affine coherence property that lets long videos be inferred window-by-window with no temporal alignment machinery and zero scale drift. The result is SOTA zero-shot video depth on benchmarks while using 163× less task-specific data than leading baselines, with the full training suite open-sourced.

Key claims

Existing video depth estimation faces a fundamental trade-off: generative depth models hallucinate geometry stochastically and exhibit scale drift, while discriminative depth models need massive labeled datasets to disambiguate semantics [§Abstract, §1].
A pre-trained video diffusion model can be turned into a deterministic single-pass depth regressor — the first such framework — by adapting the diffusion backbone rather than running it as a generative sampler [§Abstract, §1].
The diffusion timestep, normally used to schedule noise during sampling, can be repurposed at a fixed value as a “structural anchor” that balances global stability against high-frequency detail in the regressed depth [§Method (i)].
Latent manifold rectification (LMR) — a differential-constraint loss applied in latent space — counteracts the over-smoothing that direct regression of latents tends to produce, recovering sharp boundaries and coherent motion [§Method (ii)].
The framework has an inherent global affine coherence property: inter-window depth divergence is bounded, so long video inputs can be inferred window-by-window and stitched without temporal-alignment heuristics, yielding flicker-free infinite-length depth [§Method (iii), §Results].
DVD reaches SOTA zero-shot performance across video-depth benchmarks while using 163× less task-specific training data than leading baselines [§Abstract, §Experiments].
Author-reported headline: 5.5 AbsRel on ScanNet, more data-efficient than baselines, no temporal flicker, infinite-length inference with zero scale drift [tweet thread; §Experiments].
The full pipeline — training suite, code, and models — is released to the open-source community [§Abstract; project page; GitHub EnVision-Research/DVD].

Method

DVD starts from a pre-trained video diffusion model (Wan 2.1 in the released implementation) and converts it from a stochastic generator into a deterministic per-pixel depth regressor. The first design choice is to fix the diffusion timestep at a single value during inference and use it as a conditioning anchor on the backbone — the timestep embedding still flows through the network, but it no longer denotes a step in a noise schedule; it denotes the desired balance between low-frequency global structure and high-frequency detail. Training optimizes this fixed-timestep network to map an RGB video latent to a depth latent in a single pass.

The second design choice is latent manifold rectification (LMR). Regressing latents directly against a depth target tends to over-smooth — the network finds a low-loss “average” latent that decodes to a blurry depth map. LMR adds a differential constraint in latent space (along spatial and temporal axes) that forces the regressed latents to preserve the local structure of the pre-trained latent manifold, restoring sharp boundaries and coherent motion. The third design choice exploits an empirical property the authors name global affine coherence: when DVD is run on overlapping temporal windows of a long video, the per-window predictions differ from each other by an approximately affine (scale + shift) transform, so a single global alignment step suffices to stitch them — no recurrent state, no per-frame alignment search, no scale drift. This is what enables “infinite-length” inference with no temporal flicker.

Results

The paper reports SOTA zero-shot performance across standard video-depth benchmarks. Headline numbers shared by the authors:

ScanNet AbsRel 5.5 (lower is better) — author-reported SOTA [tweet; §Experiments].
Data efficiency: the leading baselines require ~163× more task-specific data than DVD to reach comparable accuracy [§Abstract, §Experiments].
Temporal stability: no temporal flicker, and inter-window predictions are bounded by an affine transform — long-video inference with zero scale drift [§Method (iii), §Experiments].

Qualitative side-by-side video comparisons on the project page show DVD vs. Video Depth Anything; the project page demos overlay synchronized depth predictions and emphasize boundary sharpness and absence of frame-to-frame flicker.

Why it’s interesting

DVD is the cleanest articulation on file of a perception-side counterargument to The flavor of the bitter lesson for computer vision: rather than treating video-generative pre-training as the substrate that should replace explicit depth estimation, DVD treats it as the prior that should fuel explicit depth estimation more data-efficiently. The provocation in the accompanying tweet — “video diffusion models are just overqualified depth estimators” — is the opposite framing of Sitzmann’s “depth and pose are dispensable intermediates”; both can be read as evidence that a generative video model has internalized geometry, but they draw incompatible normative conclusions about what to do with that fact.

It also pairs with Track4World: Feedforward World-Centric Dense 3D Tracking of All Pixels (same Slack sharer @wildmindai, two days apart): both papers adapt a feed-forward backbone to a dense per-pixel perception task. Track4World uses a VGGT-style geometric backbone and adds a 2D-3D scene-flow head; DVD uses a video generative backbone and adds a depth regression head. The pair is a clean A/B on which family of foundation models — geometric-feed-forward vs. video-generative — supplies the better prior for depth-class tasks. DVD’s 163× data efficiency claim, if it holds, is a strong vote for the generative side, at least when the task admits a per-pixel-regression head.

Within the existing wiki, DVD is also one of the first filed cases of World Foundation Models being repurposed as a perception backbone rather than as a generation, prediction, or reward-model backbone. That widens the “WFM as backbone” pattern catalogued under NVIDIA Unveils New Open Models, Data and Tools to Advance AI Across Every Industry (Cosmos as VLA backbone) and Inference-time Physics Alignment of Video Generative Models with Latent World Models (VJEPA as reward model) with a third role: WFM as a single-shot dense-prediction backbone.