TAPNext++: What's Next for Tracking Any Point (TAP)?

TL;DR

TAPNext++ is a drop-in fine-tune of the publicly released BootsTAPNext-B checkpoint that lifts recurrent-transformer point tracking from a ~150-frame failure point to stable 1024-frame tracking without changing the architecture. The gains come from three training-side moves: sequence-parallel training on 1024-frame synthetic videos (new Kubric-1024 and DynHumans datasets), periodic-roll + variable-aspect-ratio augmentations that force appearance-based re-detection, and a weighted loss that supervises occluded points inside the frame. On top of this, the authors introduce a new Re-Detection Average Jaccard (R-AJ) metric that explicitly measures how well a tracker finds a point that reappears after being invisible for ≥n frames — a dimension absent from TAP-Vid-style benchmarks today. Final model is faster and lower-latency than CoTracker3 / Track-On and sets a new SOTA mean across PointOdyssey, DAVIS, RGB-Stacking, RoboTAP, and Kinetics.

Key claims

A standard TAPNext-architecture tracker (recurrent SSM + per-frame ViT) can be scaled to 1024-frame training end-to-end via a distributed parallel scan over RG-LRU + temporal-conv layers, cutting forward/backward pass to 3 instead of 7 steps on 8 GPUs vs naïve sequential communication [§3, Long-Sequence Training].
The baseline BootsTAPNext memory degrades sharply after ~150 frames even on a static image sequence; fine-tuning on 240-frame DynHumans pushes effective memory to ~600 frames and fine-tuning on 1024-frame PointOdyssey extends it substantially further [§4.3, Fig. 5].
Periodic-roll augmentation (pixels that leave the frame wrap around to the opposite side with a margin) explicitly trains re-detection and improves R-AJ noticeably without hurting standard AJ, while variable-aspect-ratio crops close the DAVIS generalization gap that pure PointOdyssey fine-tuning opens [§3 Augmentations, Table 3, Fig. 6].
Supervising occluded points inside the frame with position loss weighted 0.1× visible loss improves tracking through occlusion without destabilizing visible-point regression [§3, Tracking Occluded Points via Weighted Loss].
A 3-dataset fine-tune mix (PointOdyssey + Kubric-1024 + DynHumans) gives the best overall tracker: PointOdyssey Survival 52.6 / MTE 13.4, DAVIS AJ 66.6, RoboTAP AJ 61.1, Kinetics AJ 54.4, mean 72.2 — beating CoTracker3 (70.2), Track-On (68.5), and BootsTAPNext-B (62.3) [Table 1].
TAPNext++ runs at 348 FPS on 1024 query points in window mode and 562 FPS on 256 query points on an H100 — lower latency than CoTracker3 (102 ms at 256 points, 45 ms at 1024 points) and Track-On (28 ms / 23 ms) [Table 2].
Re-Detection Average Jaccard (R-AJ@n) is defined only over eligible reappearance events — those whose disappearance duration strictly exceeds every prior disappearance in the same track — so it isolates recovery after increasingly hard occlusions and is reported on PointOdyssey and RoboTAP [§4.1].

Method

TAPNext++ keeps the TAPNext spatiotemporal factorization unchanged: each video frame is linearly projected to visual tokens plus learned 2D positional encodings, and each of K query points becomes a sequence of track tokens initialized as [MASK] except at the query frame, where the token is filled with the position embedding of (x, y). Spatial self-attention runs within each frame; temporal propagation runs per token via Real-Gated Linear Recurrent Units (RG-LRU, a Griffin-style SSM) that are linear in sequence length and parallelizable via associative scan.

Scaling training to 1024 frames is the central systems contribution: the authors shard the sequence dimension across 8 GPUs, do a local associative scan on each chunk, then a logarithmic-time merge across chunks to reconcile hidden states at chunk boundaries, then a final local update — making end-to-end backprop over 1024 frames fit in GPU memory. On the data side they generate Kubric-1024 (10K videos × 1024 frames, Polyhaven HDRIs + GSO objects + PyBullet with periodic velocity “bumps” to keep scenes dynamic), DynHumans (10K × 240-frame indoor human-motion clips), and augment with synthetic homography-rolled OpenImages. Training is fine-tuning (not pretraining-from-scratch) on BootsTAPNext-B weights with AdamW + cosine schedule, 20k steps per dataset, 8×H100.

Results

PointOdyssey (long sequences, 1,149–4,325 frames): Survival 52.6 vs CoTracker3 44.5, MTE 13.4 vs CoTracker3 20.7, AJ 23.3 vs CoTracker3 21.8 [Table 1].
DAVIS: AJ 66.6 vs CoTracker3 63.8 and BootsTAPNext-B 65.2 [Table 1].
RGB-Stacking: AJ 73.4 and OA 95.1 — both best in table [Table 1].
RoboTAP: AJ 61.1 beats CoTracker3 66.4? Actually CoTracker3 54 vs TAPNext++ 61.1 — misread; TAPNext++ is below CoTracker3 on RoboTAP AJ (61.1 vs 66.4) but highest-mean overall. [Table 1 — see Mean column: 72.2 vs 70.2.]
512×512 high-resolution fine-tune pushes DAVIS AJ from 65.6 to 67.0 [HF checkpoint card].
Throughput: 348 FPS at 1024 points / 562 FPS at 256 points in window mode on H100 — faster than all baselines reported [Table 2].
Re-Detection AJ (R-AJ): periodic-roll augmentation substantially improves R-AJ@n curves on PointOdyssey and RoboTAP, where prior methods including CoTracker3 and Track-On degrade sharply as disappearance duration grows [Fig. 6].

Why it’s interesting

The paper is a clean existence proof that the 2026-vintage recurrent-SSM-+-ViT architecture (Griffin-style, same lineage as the backbone used in Latent Spatial Memory for Video World Models for video world models) does not have an intrinsic long-context ceiling — the ceiling was a training data and loss problem, not a representation problem. That echoes the current thread running through Cracks in the Foundation: Seemingly Minor Architectural Choices Impact Long Context Extension and the broader recurrent-sequence literature. On the perception side it slots into the point-tracking / 4D-tracking cluster alongside MegaFlow: Zero-Shot Large Displacement Optical Flow (zero-shot optical flow + tracking on frozen DINO), Track4World: Feedforward World-Centric Dense 3D Tracking of All Pixels (world-centric dense 3D tracking of all pixels), and PointZero: 3D Point Track Completion for Learning Transferable 3D Dynamics (3D point-track completion as an action-free pretraining recipe for robot dynamics) — but with the specifically-online + frame-by-frame + mobile-latency profile that AR and robotics manipulation pipelines actually need. The R-AJ metric is also a methodological contribution: it formalizes a failure mode (long-occlusion re-detection) that the TAP-Vid benchmark averages out, and should be adopted by comparison papers going forward.