TrackEverything: Long Horizon Dense Tracking via De-Duplicating 3D Scene Representations

TL;DR

TrackEverything (CMU + Meta Reality Labs) is a 3D point tracker that breaks the long-standing sparse-vs-dense tradeoff by representing videos as persistent 3D scene tracks in world coordinates rather than per-frame 2D pixels. A sliding-window architecture unprojects per-frame DINOv3-small features into a world-coordinate feature cloud (using VGGT-style pointmaps), then at each window boundary voxelizes co-located tracks to merge physical surfaces that have already been observed — so compute scales with unique scene geometry, not frame count. It is reportedly the first 3D tracker to track all visible points across 1000+ frame videos within 40GB of GPU memory, and on TAPVid-3D it beats every open-source all-frame dense 3D tracker by >20% APD on short clips while matching sparse-tracker SOTA on long sequences despite tracking orders of magnitude more points. Code + weights were released on 2026-10-20 (the subject of Paul’s Slack message); the paper itself was posted on arXiv on 2026-09-24.

Key claims

Dense point tracking has a fundamental sparse-vs-dense tradeoff: sparse trackers scale to long horizons but miss most of the scene, while dense trackers either only track points visible in the first frame (DELTA, AllTracker, ST4RTrack, Any4D) or exhaust GPU memory beyond ~48–64 frames when tracking all points (VDPM, D4RT, TraceAnything) [§2].
Representing video in a persistent world-coordinate 3D scene space decouples model complexity from video duration — complexity instead scales with unique physical geometry [§1, §3].
Voxelization-based de-duplication at sliding-window boundaries merges co-located tracks across frames: because two physical entities cannot occupy the same voxel at the same target timestep, re-observations of the same surface are collapsed into one canonical track [§3.3].
An endpoint refiner / trajectory refiner decomposition concentrates dense trajectory decoding on only the dynamic subset of points, while static points trivially retain world-coordinate position — avoiding wasted computation on the typically-large stationary majority [§3.2].
3D WAFT (a 3D extension of warp-aligned feature transforms) replaces memory-prohibitive 4D correlation volumes with bilinear-sampled source + target feature pairs, giving the transformer an implicit template-matching signal [§3.2.1].
TrackEverything is the first 3D tracker to track all visible points across 1000+ frame videos within 40 GB of GPU memory [§1, §4.2].
On TAPVid-3D 48-frame clips, TrackEverything outperforms all open-source all-frame dense 3D trackers (VDPM) by >20% APD-P averaged across ADT / DriveTrack / PStudio [Table 1].
On full-length sequences, TrackEverything remains competitive with sparse 3D trackers (SpatialTracker-v2, TAPIP-3D, CoTracker3-3D) despite tracking orders of magnitude more points, and is the only method that scales all-point tracking past 1000 frames [§4.2, Fig. 3].
The model has 61M total parameters (41M trainable; frozen 20M DINOv3-small backbone), trained on only three synthetic datasets — Kubric, PointOdyssey, Dynamic Replica — vs ~17-dataset mixes used by SpatialTracker-v2 and the 1B+-param D4RT [§3.4].

Method

Given an RGB video and per-frame 3D pointmaps in world coordinates (from depth sensors or feedforward reconstruction models like VGGT-µ), TrackEverything processes video in non-overlapping sliding windows of T frames. Within each window: (1) a frozen DINOv3-small backbone + trainable ViT adapter encodes each frame into a feature map, which is unprojected into a world-coordinate feature cloud using the pointmaps, then per-frame voxelized to collapse within-frame redundancy. (2) An endpoint refiner — a stack of multi-head self-attention layers over the full feature cloud — iteratively updates each point’s estimated 3D position at the window boundary and simultaneously classifies it as static or dynamic. Before each pass, 3D WAFT samples both a source feature at the point’s birth location and a target feature at its current estimated position, concatenating them with the main feature; template-matching between the two gives the attention layer an implicit correspondence signal. (3) For points classified as dynamic, a trajectory refiner decodes dense within-window trajectories using alternating within-track temporal self-attention and cross-attention from a [cls] token to the full feature cloud. Static points trivially retain their world-coordinate positions. (4) At each window boundary, the feature cloud is voxelized at the updated 3D positions to merge co-located points — the central de-duplication mechanism — yielding a compact track set for the next window alongside any newly revealed scene content. The iterative loop (endpoint update → trajectory update) runs for K iterations within each window, with WAFT features and position embeddings re-sampled at each updated estimate.

Results

TAPVid-3D (48-frame clips, APD-P averaged over ADT / DriveTrack / PStudio): TrackEverything beats VDPM by over 20% APD on average among all-frame dense trackers [Table 1]. D4RT reports higher numbers on DriveTrack and PStudio but uses 20× the parameters, trains on orders of magnitude more public + private data, has no public code/weights, and does not scale beyond 48 frames.
Full-length TAPVid-3D sequences: TrackEverything is competitive with sparse 3D trackers (SpatialTracker-v2, TAPIP-3D, CoTracker3-3D) despite tracking orders of magnitude more points per frame [Table 1].
Scaling with video length (PointOdyssey, L40S-46G): all baselines (VDPM, Any4D, DeltaV2, SpatialTracker-v2) OOM beyond ~96 frames; TrackEverything is the only method that completes 1000+ frame videos within 40 GB [Fig. 3a].
Model size and training data: 61M total params (20M frozen DINOv3-small + 41M trainable), 3 synthetic datasets (Kubric + PointOdyssey + Dynamic Replica). Trained for 100k iterations on 32-frame clips + 300k iterations on 32/64-frame mix on 8× L40S-46G [§3.4].
Point cloud geometry source: external VGGT-µ pointmaps are provided to TrackEverything and to all directly-compared baselines (SpatialTracker-v2, TAPIP-3D, CoTracker3, Any4D, DeltaV2) to isolate the tracking contribution from geometry quality [§4.1].

Why it’s interesting

The paper lands cleanly in the dense-point-tracking cluster already on file as the strongest all-points-all-frames at long horizon data point to date. It is the perception-side counterpart to Track4World: Feedforward World-Centric Dense 3D Tracking of All Pixels — both build on VGGT-style geometric foundations and emit world-coordinate dense 3D tracks, but Track4World fuses pairwise feed-forward scene-flow between arbitrary frame pairs (short-to-medium horizon), while TrackEverything introduces a persistent 3D state with cross-window voxelization that explicitly makes complexity scale with scene geometry rather than frame count — the natural next step for the 1000+ frame regime that Track4World does not reach. It is also complementary to TAPNext++: What's Next for Tracking Any Point (TAP)? TAPNext++, which solves long-horizon tracking in the recurrent-SSM 2D-online setting; TAPNext++ and TrackEverything together bracket the two obvious scaling strategies — RNN state vs persistent 3D scene — for the long-horizon tracking frontier. Pointwise, TrackEverything’s voxelization-based de-duplication is a concrete answer to the “where does context go when you need all-point tracking over long horizons” question that PointZero: 3D Point Track Completion for Learning Transferable 3D Dynamics PointZero sidesteps by only predicting 3D track completions from a sparse set of visible observations. The Slack post flags the delayed code+weights release (paper on arXiv 2026-09-24, repo made public 2026-10-20) — directly relevant to downstream VLA / world-model stacks that want dense world-coordinate point tracks as a supervision signal.