Forecasting Motion in the Wild

TL;DR

Proposes dense 2D point trajectories — extracted with BootsTAPIR and stabilized via RANSAC homographies — as the visual token for behavior forecasting, sitting in a middle ground between raw-pixel video generation (too entangled with appearance) and parameterized 3D models (too category-specific). A DiT denoises an unordered set of per-track tokens, each carrying frozen DINOv3 image features at the track’s start point plus reparameterized velocity targets and an occlusion scalar. Curates MammalMotion (~300 hours of in-the-wild animal video filtered from MammalNet, with shot detection + camera-motion compensation) and reports the first large-scale evidence that animal motion magnitudes are log-normally distributed across many species. Outperforms ATM, Track2Act, and a zero-shot WHN baseline, and produces qualitatively physical motion for rare species, robots, and butterflies.

Key claims

Point trajectories are positioned as a third representation for behavior, alongside pixel-space video generation (universal but entangles appearance/lighting/camera with dynamics, expensive, data-inefficient) and parameterized 3D models (compact, physical, but category-specific to a small set of carefully modeled species) [§1].
Each transformer token corresponds to a full point trajectory, not a (point, time) cell — concatenating frozen DINOv3 features at the track’s start, an embedded velocity+occlusion history, and the noisy diffusion target along the channel dimension, so the model can reason jointly about a single point’s trajectory and across nearby points via attention while being invariant to track ordering [§3.2, Fig. 2].
The diffusion target is reparameterized away from raw coordinates to scaled velocity differences with linearly interpolated values across occluded points; the discrete occlusion indicator is denoised alongside as a scalar even though it’s discrete [§3.1].
Position encodings are derived from each track’s initial 2D location (sinusoidal) rather than from a sequence index, because tracks are not on any grid and have no natural order [§3.2].
DDIM with η=0 and 100 sampling steps gives a 10× speedup over 1000 training steps with minimal quality degradation; after sampling in velocity space the absolute coordinates are recovered by cumulative summation [§3.3].
The data pipeline isolates animal-only motion from in-the-wild video by (a) dense BootsTAPIR tracking, (b) shot detection via smoothly-moving-point thresholds, (c) Grounding-DINO + VideoSAM for animal masks, and (d) RANSAC homography on background tracks to estimate per-frame camera motion, approximating a world-coordinates frame without explicit 3D reconstruction [§4].
MammalMotion is filtered down from the 539-hour MammalNet dataset to ~300 hours, with stratified sampling across species and motion-magnitude bins (<0.5px, 0.5–N px, >N px frame-to-frame) for evaluation [§5.1].
The first large-scale, taxonomically-broad measurement finds animal motion magnitudes are log-normally distributed rather than power-law, consistent with prior species-specific findings (Lévy flights, foraging in rats, spontaneous behavior); interpreted as evidence that motion magnitude is a multiplicative product of independent factors [§4, Fig. 4].
Beats non-learned (no-motion, constant-velocity, oracle-velocity), ATM (regression-based track transformer), Track2Act (DiT on absolute XY with ResNet visual features + AdaLN, the closest prior architecture), and a zero-shot What-Happens-Next baseline on the Panthera and all-species splits across ADE, FDE, PWT, FD-velocity, FD-acceleration, FVMD, and VMD [§5.3, §6].
Predicts physically plausible motion on out-of-distribution inputs that the training data never represents — a Lego robot and a butterfly — suggesting trajectory tokens generalize beyond mammalian morphology [§6, Fig. 7].
A side-by-side qualitative comparison with Stable Video Diffusion argues that because pixel video models must simulate texture, lighting, and camera in addition to motion, they tend to produce less physically realistic motion than a model that diffuses trajectory tokens directly — extending the [7]-style finding from synthetic rigid Kubric objects to in-the-wild non-rigid animals [§6, Fig. 8].

Method

The model is a standard DiT [Peebles & Xie] denoising an unordered set of per-track tokens; the set size varies per example. For the k-th track, the input token is constructed by concatenating along the channel dimension: (i) the DINOv3 feature bilinearly sampled at the track’s initial 2D location inside the animal’s expanded bounding-box crop, (ii) a sinusoidal embedding of the clean velocity history $V_{\text{hist}}$ scaled by $\alpha$ and the clean occlusion history $O_{\text{hist}}$ scaled by $\beta$ , (iii) the noisy diffusion target — scaled velocity differences $V_{\text{noisy}}$ and a scalar occlusion $O_{\text{noisy}}$ . Each token is projected to model dim and gets a sinusoidal positional encoding derived from the starting xy of that track, not from a sequence index. Adaptive layer norm injects the global conditioning — diffusion timestep $t$ and (optionally) the average displacement vector $\Delta$ , which lets a user prompt “more motion / less motion / motion in this direction” at sample time. Training uses an L1 denoising loss on the velocity-space diffusion target. The pipeline matters as much as the model: BootsTAPIR for tracks, shot detection via smoothly-moving-point counts, Grounding-DINO + VideoSAM for animal masks, RANSAC over background tracks for a per-frame homography that approximates camera motion (no full 3D reconstruction), and bias toward thin structures so legs and tails are represented in the sparse track set.

Results

The headline qualitative claim is on diverse non-rigid motion: lion gaits with correct articulation, jaguar grooming, alpaca grazing, with sample-to-sample diversity from different seeds producing different head directions and grooming frequencies. Optional displacement conditioning steers samples toward “more / less / different-direction” motion plausibly. Out-of-distribution generalization to a butterfly and a Lego robot is reported qualitatively (Fig. 7) — neither in MammalNet’s training distribution. Quantitatively (best-of-N), the model beats no-motion, constant-velocity, the oracle-velocity baseline, ATM, Track2Act, and zero-shot WHN across Panthera and all-species splits on ADE/FDE/PWT (example-level) plus FD-velocity, FD-acceleration, FVMD, and VMD (distribution-level); exact numbers and per-split tables are in §6 / appendix and not reproduced here. The side-by-side qualitative comparison against Stable Video Diffusion (Fig. 8) is presented as evidence that trajectory tokens produce more physically realistic motion than pixel video at comparable or lower compute, extending the rigid-Kubric finding of [7] to in-the-wild non-rigid animals.

Why it’s interesting

This is the most direct on-wiki challenge to the pixel-side framing of World Foundation Models — Neerja Thakkar’s framing question on the tweet thread (“3D, pixels, or something else?”) names the same axis The flavor of the bitter lesson for computer vision argues over: Sitzmann thinks generative-rollout pixel WFMs should replace explicit 3D, but this paper argues the opposing case — that pixels are too entangled and 3D is too category-specific, and that a mid-level tracks representation is the right pre-training target for behavior. Mechanically it inherits the Track2Act recipe (point tracks as DiT tokens, used in robotics) and pushes it to the in-the-wild non-rigid domain, complementing MegaFlow: Zero-Shot Large Displacement Optical Flow (which generates dense optical flow as the target, not future trajectories) and Track4World: Feedforward World-Centric Dense 3D Tracking of All Pixels (feed-forward world-centric dense 3D tracking, a perception sibling to this paper’s forecasting). The log-normal animal-motion distribution finding is independently a useful empirical anchor for any future motion-prior work. The DINOv3-as-frozen-feature pattern echoes V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning — both lean on frozen self-supervised vision backbones as the appearance prior — but here the predictive head is a diffusion model over trajectories rather than a JEPA-style latent predictor.