Depth Anything 3: Recovering the Visual Space from Any Views

TL;DR

Depth Anything 3 (DA3) is ByteDance Seed’s any-view geometry foundation model: a single plain ViT (vanilla DINOv2 backbone) plus a Dual-DPT head that jointly predicts pixel-aligned depth and ray maps from one or many images, with or without known camera poses. The “minimal modeling” thesis is the contribution — the authors argue depth + ray is a sufficient prediction target (no point maps, no separate pose head) and that a stock transformer with input-adaptive cross-view self-attention suffices (no DUSt3R/VGGT-style bespoke architecture). On a new 5-dataset visual-geometry benchmark DA3 beats VGGT by 35.7% on camera pose and 23.6% on geometric accuracy averaged, while also beating DA2 on monocular depth — and it’s trained exclusively on public academic data.

Key claims

A depth-ray representation (per-pixel depth + a per-pixel ray map encoding origin and direction) is a minimal sufficient prediction target for joint multi-view depth and pose, outperforming point-map or multi-task redundant targets at fixed compute [§3.1, Table 6].
A single plain ViT (DINOv2 init) suffices as backbone; cross-view reasoning is added without architectural changes by alternating within-view and cross-view self-attention via input token rearrangement in the last L₂ of 24 layers (best L₂=8) [§3.2, Table 7].
A Dual-DPT head shares reassembly modules but uses separate fusion layers for depth vs ray, giving strong inter-task interaction without redundant intermediates [§3.2, Table 6].
Camera-pose conditioning is injected via a single per-view camera token from a lightweight MLP (or a learned placeholder when poses are absent); the same model handles posed and unposed inputs [§3.2].
A teacher–student paradigm using a DA2-style monocular depth teacher (trained only on synthetic data, predicting scale-shift-invariant exponential depth) generates high-quality pseudo-depth aligned to noisy real-world depth via RANSAC least squares — this is what makes mixed real+synthetic training tractable [§3.3, §4].
On the authors’ new visual-geometry benchmark (5 datasets, 89+ scenes), DA3 sets SOTA on 18/20 settings, surpassing VGGT by an average 35.7% on camera-pose accuracy and 23.6% on geometric accuracy [§1, §Experiments].
On monocular relative depth, the DA3-Mono variant outperforms DA2 on standard benchmarks while being trained only on unlabeled images supervised by the DA3 teacher [§4.3, Table 10].
Fine-tuning the geometry foundation with an additional DPT head for pixel-aligned 3D Gaussians produces a feed-forward NVS model that beats specialized FF-NVS models like DepthSplat, and FF-NVS quality correlates directly with the upstream geometric accuracy [§1, §Feed-Forward NVS].
All models trained exclusively on public academic datasets; checkpoints, code, and a benchmark evaluation pipeline are released, including monocular relative, monocular metric, any-view, and any-view+Gaussian variants [§3.4, model cards in repo].

Method

DA3 takes N input images and outputs, per pixel per view, a depth map and a 6-channel ray map (3 channels for ray origin in world frame, 3 for unnormalized ray direction whose magnitude preserves projection scale). A 3D point is then trivially recovered as origin + depth * direction, so depth + ray together define a consistent world point cloud across all views without ever predicting a rotation matrix. Camera intrinsics and extrinsics are recovered from the ray map at inference by averaging ray origins for the center and solving a homography between canonical and predicted ray directions via DLT + RQ decomposition; a small auxiliary “camera head” operating on per-view camera tokens also directly emits FOV + quaternion + translation as a cheap shortcut [§3.1].

Architecturally, the backbone is a 24-block DINOv2 ViT split into L₁=16 within-view layers and L₂=8 layers that alternate cross-view and within-view attention. Cross-view attention is implemented purely by rearranging tokens — every token across all N views attends jointly — so the same weights handle N=1 (monocular) through N=18 (multi-view/video) without any architectural switch. A camera token (MLP-projected from intrinsics+extrinsics when known, or a shared learnable token otherwise) is prepended to each view’s patch tokens and participates in all attention ops, providing optional pose conditioning. The Dual-DPT head shares reassembly modules across both prediction streams but uses separate fusion stacks for depth and ray, ending in two output layers [§3.2].

Training uses a teacher–student paradigm. The DA3 Teacher is a DA2-style monocular depth model retrained on a vastly expanded synthetic corpus (Hypersim, TartanAir, IRS, vKITTI2, BlendedMVS, SPRING, MVSSynth, UnrealStereo4K, MatrixCity, Objaverse, Trellis, OmniObject, …) with three changes vs DA2: (i) scale-shift-invariant depth not disparity, (ii) exponential depth representation for better near-camera resolution, (iii) a distance-weighted surface-normal loss plus joint sky/object mask prediction. The teacher’s dense relative-depth output is aligned to noisy real-world ground truth via RANSAC scale-shift estimation, then used to supervise DA3 from step 120k onward — supervision transitions from raw GT to teacher labels. DA3 itself is trained for 200k steps on 128 H100s, peak LR 5e-4, base resolution 504×504 with multi-resolution sampling and 2–18 view counts; pose conditioning is randomly activated with 20% probability so a single checkpoint serves both posed and unposed regimes [§3.3, §3.4, §4].

Results

Visual-geometry benchmark (HiRoom, ETH3D, DTU, 7Scenes, ScanNet++; 89+ scenes): DA3 sets SOTA on 18/20 settings; average +35.7% camera-pose accuracy over VGGT and +23.6% geometric accuracy [§1].
DA3NESTED-GIANT-LARGE AUC3 with ray-head pose recovery vs camera-head: HiRoom 84.4 / 80.3, ETH3D 52.6 / 48.4, DTU 93.9 / 94.1, 7Scenes 29.5 / 28.5, ScanNet++ 89.4 / 85.0 — ray-head decoding is generally more accurate but slower; cam-head is the default [FAQ, repo README].
Monocular depth: DA3-Mono-Large beats DA2 on standard monocular benchmarks despite being trained only on unlabeled images via teacher pseudo-labels [§4.3, Table 10].
Feed-forward NVS: Fine-tuning DA3 with a pixel-aligned 3D-Gaussian DPT head beats specialized FF-NVS models like DepthSplat on a 160+ scene benchmark; geometry accuracy correlates directly with NVS quality [§1, §FF-NVS].
Model family (1.40B → 0.08B): Nested-Giant-Large (1.40B), Giant (1.15B), Large (0.35B), Base (0.12B), Small (0.08B), plus DA3Metric-Large for monocular metric depth in meters and DA3Mono-Large for monocular relative depth; all under Apache 2.0 or CC-BY-NC 4.0 [Model cards].
DA3-Streaming: Sliding-window streaming inference for ultra-long video sequences runs under 12GB GPU memory, released 2025-12-11 [Repo news].
DA3-1.1 retraining: The Giant, Large, and Nested-Giant-Large checkpoints were retrained after fixing a training bug; the -1.1 suffixed models are preferred, with substantially better performance on street scenes [Model cards].

Why it’s interesting

This is the direct production-grade successor to DA2 and the cleanest contender filed against VGGT-Ω: Scaling Feed-Forward Reconstruction for Static and Dynamic Scenes on the feed-forward visual-geometry frontier. The two papers make opposite architectural bets: VGGT-Ω scales VGGT’s bespoke multi-task-with-registers transformer by replacing its DPT decoder with a single MLP head and routing inter-frame information through register tokens; DA3 keeps a stock DINOv2 backbone with token-rearrangement cross-view attention and a Dual-DPT head, and argues that depth + ray is a minimal prediction target making point maps and explicit pose heads redundant. Both report large gains over VGGT itself, which makes the design-space comparison interesting — same task, very different theories of what was load-bearing in VGGT. DA3 is also a natural backbone candidate for the feed-forward Gaussian reconstruction cluster: like Track4World: Feedforward World-Centric Dense 3D Tracking of All Pixels (a VGGT-style backbone repurposed for dense scene-flow tracking) and NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos (VGGT “Gaussianized” into 4DGS), DA3 demonstrates the geometry-foundation-plus-small-task-head recipe; in fact Track4World already reports DA3 as a competitive Bonn-pose baseline. Pair this with DVD: Deterministic Video Depth Estimation with Generative Priors‘s opposite framing — video diffusion as the depth prior rather than a dedicated geometry transformer — and the wiki has a clean A/B/C: dedicated geometry transformer (DA3, VGGT-Ω), generic geometry foundation + task head (Track4World, NeoVerse), and video-generative prior repurposed for perception (DVD).