D4RT: Efficiently Reconstructing Dynamic Scenes One D4RT at a Time

TL;DR

D4RT is a Google DeepMind feed-forward transformer that jointly infers depth, spatio-temporal correspondence (3D tracks), and full camera parameters from a single monocular video, all from a unified architecture. The key design move is a queryable decoding interface: a global self-attention encoder produces a latent Global Scene Representation, and a single lightweight decoder is independently queried for the 3D position of any 2D point (u, v) at source timestep t_src as observed at target timestep t_tgt in camera coordinate frame t_cam — sidestepping dense per-frame decoders and task-specific heads. The same network covers 3D tracking, 3D reconstruction (depth + camera projection), and dense all-pixels tracking in world coordinates. The paper claims state-of-the-art across a wide spectrum of 4D reconstruction tasks.

Key claims

• A single unified transformer can jointly produce depth, dense 2D-3D correspondence, and camera parameters from a single video in a single forward pass [§Abstract].
• The queryable-decoder interface — probing the 3D position of any (u, v, t_src, t_tgt, t_cam) tuple independently — replaces both heavy dense per-frame decoders and the complexity of multiple task-specific decoders [§Method].
• Local frame patches centered on (u, v) are embedded into the query alongside the global scene representation, providing extra spatial context to the decoder [§Method].
• The same model exposes three downstream regimes without architectural change: (i) sparse 3D tracking in local camera coordinates, (ii) 3D reconstruction by projecting depth via camera pose (no correspondences; dynamic objects are deduplicated), and (iii) dense all-pixels 3D tracking in world coordinates [§Tasks].
• D4RT sets a new state of the art across a wide spectrum of 4D reconstruction tasks [§Abstract].

Method

D4RT’s architecture is a two-stage feed-forward transformer. A global self-attention encoder ingests an input video and produces a latent Global Scene Representation — a set-style latent intended to be the unified geometric prior, in the style of SRT (Scene Representation Transformer) and the VGGT family. A lightweight decoder then accepts queries of the form (2D point (u, v), source timestep t_src, target timestep t_tgt, camera coordinate frame t_cam) plus a local RGB patch embedding centered on (u, v), and returns the predicted 3D position. Decoupling the decoder from the encoder, and making the decoder an independently-queryable function over space and time, is what unlocks fast inference for sparse queries and converts dense reconstruction into a parallel batch of independent queries.

The same query interface handles all three reported regimes: sparse 3D tracking returns 3D positions for a few selected pixels in selected frames; depth-based reconstruction queries every pixel at (t_src = t_tgt) and projects through the predicted camera pose; dense all-pixels world-coordinate tracking queries every pixel across every (t_src, t_tgt) pair and uses the camera pose to align into a global frame.

Results

The project page does not disclose numerical tables — it claims new state of the art “across a wide spectrum of 4D reconstruction tasks” and showcases qualitative results on 3D tracking, 3D reconstruction, and all-pixels-tracking. The interactive viewer adapts the WebGL splat viewer for 4D visualization. Specific benchmark numbers, ablations, and training-data scale are deferred to the (unreleased) arXiv preprint cited as zhang2025d4rt.

Why it’s interesting

D4RT lands in the same architectural neighborhood as VGGT-Ω: Scaling Feed-Forward Reconstruction for Static and Dynamic Scenes and Track4World: Feedforward World-Centric Dense 3D Tracking of All Pixels — feed-forward transformer over a video that emits geometry — but takes a different stance on the decoder: instead of VGGT-Ω’s compressed register tokens + dense MLP head or Track4World’s sparse-to-dense 2D-3D correlation decoder, D4RT exposes a single SRT-style queryable decoder that you call once per (point, time, frame) tuple. That recipe is closer to a Neural Field’s query interface than to a dense per-frame head, and it inherits the same scaling property: inference cost is linear in the number of queries you actually need, not in the resolution of the output. Matt’s note links this to Luma’s Ray3 stack — the “all-pixels world-coordinate 3D tracking” regime is exactly the perception primitive a generative video / world model needs to consume to do explicit 4D control, and a feed-forward variant that ships with depth + correspondence + camera in one pass closes a real preprocessing bottleneck. On the foundation-layer debate, D4RT is another datapoint against The flavor of the bitter lesson for computer vision‘s explicit-3D-will-be-dissolved argument: it doubles down on scaling a unified explicit-geometry foundation model, in the same bet as VGGT-Ω. The DeepMind authorship overlap (Sajjadi, Sukthankar, Zisserman, Ghahramani, Hadsell) overlaps with the SRT / OSRT / RUST lineage, which is a cleaner pedigree for “queryable scene representation” than the VGGT line.

See also

• Track4World: Feedforward World-Centric Dense 3D Tracking of All Pixels — same problem (feed-forward dense 3D tracks of all pixels in world coordinates), different decoder design (sparse-to-dense 2D-3D correlation vs queryable SRT decoder); a head-to-head on the same benchmarks would be illuminating.
• VGGT-Ω: Scaling Feed-Forward Reconstruction for Static and Dynamic Scenes — the other 2026 contender for “unified feed-forward geometry foundation model”; VGGT-Ω scales the register-token + dense-MLP recipe, D4RT scales the queryable-decoder recipe.
• 4D Scene Generation — concept page collecting the generative side of the 4D problem; D4RT is upstream perception that a NeoVerse/VerseCrafter-style stack could consume directly.
• NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos — “Gaussianizes” VGGT into a feed-forward 4DGS reconstructor; D4RT could play the same role under a different decoder.
• The flavor of the bitter lesson for computer vision — the position essay arguing this whole recipe is a hand-crafted intermediate the Bitter Lesson will dissolve; D4RT is a contemporary counter-bet from DeepMind.
• World Foundation Models — D4RT is a predictive world foundation model in the reconstruction sense.