tapnet — Tracking Any Point (TAP) model family and benchmarks

TL;DR

Official Google DeepMind repository bundling every model, dataset, and demo in the Tracking Any Point (TAP) line: TAP-Net, TAPIR, BootsTAPIR, RoboTAP, TAPVid-3D, TAPNext, TRAJAN, and the recent TAPNext++. All code and pre-trained checkpoints are Apache-2.0, in both JAX and PyTorch, including Colab notebooks and a GPU real-time demo for interactively selecting and tracking points. The flagship as of filing is TAPNext++ — a recurrent transformer that tracks points frame-by-frame in a fully online fashion, with 40× longer stable tracking than TAPNext, re-detection through occlusion, and a 512×512 fine-tune checkpoint on top of the 256×256 base model.

Key claims

The repo covers the full Tracking Any Point family released by DeepMind over 2022–2026: TAP-Vid and TAPVid-3D benchmarks (datasets + metrics), TAP-Net / TAPIR / BootsTAPIR / TAPNext / TAPNext++ model checkpoints, RoboTAP benchmark and clustering code, and a TRAJAN motion-embedding demo [README].
All checkpoints are released under Apache-2.0; checkpoints also mirrored on Hugging Face at google/tapnet [README, HF model card].
TAPNext++ is the authors’ “latest, most capable, fastest, yet simplest” tracker — formulates TAP as next-token prediction, fine-tuned via the BootsTAP procedure on 1024-frame synthetic sequences, and has “40× longer stable tracking performance” plus re-detection through occlusion vs TAPNext [README].
Reported TAPNext++ numbers in the repo table at 256×256: TAP-Vid-DAVIS AJ 65.6%, Kinetics AJ 53.9%, RoboTAP AJ 61.1%; 512×512 fine-tune raises DAVIS AJ to 67.0% [README checkpoint table].
Install path is a single pip line: pip install "tapnet[torch] @ git+https://github.com/google-deepmind/tapnet.git"; API is 3 lines (TAPNextPP.from_checkpoint(...) → model.track_frame(frame, query_points_xy=...) → recurrent state passed through subsequent frames) [HF model card quick-start].
Repo also ships Kubric data-generation utilities and point-track evaluation pipelines, not just inference code [README].

Method

The repo collects models that share one problem framing — given a query point (x, y, t) in a video, return per-frame (x, y, visible) across the rest of the video — but with different architectures. TAP-Net was the original CNN-feature-matching baseline. TAPIR added iterative refinement over sliding windows. BootsTAPIR bootstrapped self-supervised training on unlabeled web video. The TAPNext family (TAPNext → TAPNext++) breaks from the windowed-refinement paradigm entirely: a Griffin-style backbone with alternating ViT (spatial) and State-Space-Model (temporal) blocks propagates per-track tokens through time via a constant-sized linear-recurrent hidden state. TRAJAN is a parallel line that learns an embedding over motion trajectories (object-appearance invariant) for comparing videos, evaluating generative video model realism, etc. See TAPNext++: What's Next for Tracking Any Point (TAP)? for TAPNext++‘s architectural and training details.

Results

The flagship numbers (from the paper and HF card):

TAPNext++ @ 256×256: DAVIS AJ 65.6% / Kinetics AJ 53.9% / RoboTAP AJ 61.1%; PointOdyssey Survival 52.6, MTE 13.4 — SOTA across most TAP benchmarks.
TAPNext++ @ 512×512: DAVIS AJ 67.0%.
Throughput: 348 FPS tracking 1024 points on H100 in window mode; 562 FPS at 256 points.
Ranks: Papers-with-Code lists TAPNext++ first on PointOdyssey v1.2 Test (Query-First 256px) and third on TAP-Vid DAVIS and TAP-Vid RGB-Stacking at the same resolution.
Ecosystem: ~2k GitHub stars, 193 forks at filing; both JAX and PyTorch checkpoints maintained, with offline + online real-time demos.

Why it’s interesting

For Luma’s motion / tracking workflows this is the lowest-friction entry point to online point tracking at research-grade quality — the recurrent-SSM design is specifically what makes it tractable at 1024 points on a single GPU, which is roughly the regime any VFX/animation integration cares about. It composes well with the perception stack: TRAJAN’s trajectory embeddings are directly usable as a realism signal for generative video eval (adjacent to the metric-design arguments in The Pulse of Motion: Measuring Physical Frame Rate from Visual Dynamics and the motion-fidelity benchmarks in MRT: Masked Region Transformer for Layered Image Generation and Editing at Scale), and the TAP-Vid-3D benchmark + RoboTAP tracks are the de-facto evaluation substrate several filed robotics papers already report on. Contrast with MegaFlow: Zero-Shot Large Displacement Optical Flow (DINO-backbone + variable-length zero-shot optical flow — overlapping but window-based), Track4World: Feedforward World-Centric Dense 3D Tracking of All Pixels (world-centric 3D dense tracking — different problem, 3D-first) and PointZero: 3D Point Track Completion for Learning Transferable 3D Dynamics (3D track completion for robot dynamics pretraining — downstream consumer of TAP-style tracks).