Track4World: Feedforward World-Centric Dense 3D Tracking of All Pixels

TL;DR

Track4World is a feed-forward model that produces holistic, world-centric dense 3D tracking of every pixel from a monocular video — combining a VGGT-style ViT backbone for global scene/geometry encoding with a sparse-to-dense 2D-3D joint scene-flow decoder. Unlike prior monocular 3D trackers that either track sparse first-frame queries or rely on slow per-scene optimization, Track4World jointly predicts 2D and 3D dense flow between arbitrary frame pairs in a single forward pass, then fuses pairwise flows into long-range world-coordinate tracks. It posts large gains over RAFT-3D / Any4D / V-DPM on scene-flow (Kubric-3D EPE3D 0.15 vs 0.39 / 0.41 best baseline), and over CoTracker3 / BootsTAPIR on 2D point tracking (Kinetics AJ 59.1 vs 55.8). Notable for being the first filed paper that builds an end-to-end 3D tracker on top of a VGGT-style geometry foundation model.

Key claims

A single feed-forward model can produce world-centric dense 3D tracks for every pixel by combining a VGGT-style global-scene encoder with a sparse-to-dense 2D-3D joint flow decoder — no per-scene optimization, no first-frame query bottleneck [§Method, Fig. (a-d)].
A novel 2D-to-3D correlation scheme allows the same network to supervise and emit 2D and 3D flow jointly, which improves efficiency over independent 2D + 3D heads and enables 2D-3D joint supervision [§Method (c)].
Decoupling camera ego-motion from object motion lets the same architecture target either the camera-coordinate frame or the global world frame; world-frame tracking is reported as a separate evaluation regime in the same model [§Method, Table — World coordinate 3D tracking].
Pairwise scene-flow predictions between arbitrary timesteps can be fused into long-horizon dense 3D tracks without a recurrent state, by composing 2D-3D flows across frame pairs [§Method (d)].
On scene + optical flow (Kubric-3D val short): Abs Rel 0.0344 / δ<1.25 0.972 / EPE3D 0.154 / EPE2D 1.87 — beats RAFT-3D’s EPE3D 0.617, OpticalExpansion 0.209, POMATO 0.967, ZeroMSF 0.353, Any4D 0.391, V-DPM 0.409 and beats RAFT’s EPE2D 6.80 [Table — Kubric-3D val short].
Long-range Kubric-3D (val long) holds the same ordering with EPE3D 0.481 vs next-best Any4D 1.244 — the gap to baselines widens with longer horizons, not narrows [Table — Kubric-3D val long].
On KITTI scene flow: EPE3D 0.074 (next best OpticalExpansion 0.242, V-DPM 0.446); on BlinkVision EPE3D 0.114 vs next-best ZeroMSF 0.394 [Table — KITTI / BlinkVision].
On camera-coordinate 3D tracking averaged over PointOdyssey/ADT/PStudio/DriveTrack at L-16: average accuracy 0.571 vs POMATO 0.489, ZeroMSF 0.478, SpatialTracker* 0.400, DELTA* 0.432, STV2 0.240 (with * indicating GT intrinsics, † no bundle adjustment) [Table — Camera-coordinate 3D tracking].
On world-coordinate 3D tracking (the same datasets, L-16): average 0.564 vs V-DPM 0.514, Any4D 0.504, POMATO‡ 0.436 — Track4World is the only model claiming uniform leadership across both coordinate systems [Table — World-coordinate 3D tracking].
On 2D point tracking benchmarks: Kinetics AJ 59.1 / δ_vis 71.3 / OA 90.6 beats CoTracker3 (55.8 / 68.5 / 88.3) and BootsTAPIR (54.6 / 68.4 / 86.5); RoboTAP AJ 70.9 vs 66.4; RGB-S AJ 78.2 vs 71.7 [Table — TAP benchmarks].
On video depth + camera pose: Sintel ATE 0.119 / RTE 0.054 / RRE 0.309 is competitive with VGGT (0.167 / 0.062 / 0.490) and Pi3 (0.088 / 0.043 / 0.299); Bonn ATE 0.009 ties or beats DA3 0.010 and Pi3 0.012 — the tracking head does not degrade the underlying geometry foundation [Table — Sintel / Bonn].

Method

Given a monocular video, Track4World runs three stages in a single forward pass. (a) Input frames go through a VGGT-style ViT that produces global scene representations — geometric embeddings, dense point clouds, and per-frame camera poses. This is the same backbone family as VGGT / MapAnything / Pi3, repurposed here as the geometric prior for tracking. (b) A sparse-to-dense scene-flow decoder predicts joint 2D-3D flow between arbitrary timestep pairs. The novel ingredient is a 2D-to-3D correlation scheme that constructs the 3D correlation volume by lifting 2D feature correspondences into 3D space using the predicted geometry, instead of building a 4D 3D-3D correlation volume directly. This is what makes the decoder both efficient and amenable to joint 2D-3D supervision. (c) Pairwise flows from (b) are fused into a holistic world-centric 3D track for every pixel — by composing 2D-3D flows across frame pairs and using the decoded camera poses to align them into the global frame.

The decoupling of camera ego-motion from object motion is what lets the same network produce both camera-coordinate and world-coordinate tracks: in camera coordinates the 3D tracks include ego-motion, in world coordinates they don’t. The world-coordinate evaluation table reports the model under both regimes without any architectural change.

Results

Headline numbers (selection, lower is better unless noted):

Scene + Optical Flow, Kubric-3D val short (full table): EPE3D 0.154 vs RAFT-3D 0.617 / Any4D 0.391 / V-DPM 0.409; EPE2D 1.87 vs RAFT 6.80 / SEA-RAFT 9.58; δ<1.25 ↑ 0.972 vs 0.955 (Any4D best baseline). [Table — Kubric-3D val short]
Long-horizon Kubric-3D val long: EPE3D 0.481 vs Any4D 1.244 / V-DPM 1.262 / ZeroMSF 1.218; EPE2D 15.1 vs RAFT 51.9. [Table — Kubric-3D val long]
KITTI scene flow: Abs Rel 0.071, δ<1.25 0.957, EPE3D 0.074 — best on all eight reported scene-flow metrics. [Table — KITTI]
BlinkVision: Abs Rel 0.037, δ<1.25 0.977, EPE3D 0.114 — best on all eight. [Table — BlinkVision]
3D tracking average over PointOdyssey/ADT/PStudio/DriveTrack at L-16 / L-50: camera-coordinate 0.571 / 0.547 vs POMATO 0.489 / 0.474; world-coordinate 0.564 / 0.532 vs V-DPM 0.514 / 0.467. [Table — 3D tracking]
2D TAP: Kinetics AJ 59.1 / δ_vis 71.3 / OA 90.6; RoboTAP AJ 70.9 / δ_vis 81.8 / OA 93.3; RGB-S AJ 78.2 / δ_vis 88.5 / OA 92.3. Beats CoTracker3 on every metric on every benchmark. [Table — TAP]
Video depth + camera pose: Sintel ATE 0.119 / RTE 0.054 / RRE 0.309; Bonn ATE 0.009 / RTE 0.009 / RRE 0.604. Competitive with Pi3 / DA3, better than VGGT / MapAnything. [Table — Sintel / Bonn]

The pattern is uniform: leadership widens with task difficulty (longer horizons, more challenging benchmarks), and the underlying geometry foundation does not degrade — the model is doing tracking as well as geometry, not at the expense of it.

Why it’s interesting

Track4World plugs perception cleanly into the same VGGT backbone family that NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos “Gaussianizes” for generative 4D — both papers build on a feed-forward geometry foundation model and add a small task-specific head (bidirectional motion / Gaussian attributes in NeoVerse; 2D-3D scene-flow decoder here). The shared bet is that a frozen VGGT-style backbone is the right primitive for downstream 4D tasks, perception or generative. Compared with the explicit-4D-state generative recipes catalogued in 4D Scene Generation (VerseCrafter’s per-object Gaussian trajectories, SpaceTimePilot’s CamTime), Track4World is the perception-side counterpart: same geometric scaffold, no diffusion model, supervised on real scene-flow datasets. The world-vs-camera-coordinate split in evaluation is the cleanest articulation on file of why ego-motion decoupling matters as an explicit axis. Worth tracking (no pun) as upstream perception for the VLA / world-model stack — dense world-frame point tracks are exactly the supervision signal that The Waymo World Model: A New Frontier For Autonomous Driving Simulation-class simulators consume.