PointZero: 3D Point Track Completion for Learning Transferable 3D Dynamics

TL;DR

PointZero is an action-free pretraining recipe for 3D dynamics: given a single RGB-D observation and a small number of complete 3D point tracks as conditioning, predict future 3D tracks of all observed scene points. The authors argue the standard “world model needs robot action labels” constraint excludes web video from the pretraining pool, and that 3D-track supervision is the missing action-agnostic substrate — the labels are just 3D point tracks, obtainable from simulation or in principle from any tracked video. They contribute a 2.9M-frame synthetic dataset spanning deformable, articulated, and rigid objects, train a Perceiver-IO + denoising DiT (comparing flow matching vs JiT vs regression) on it, and then post-train the same model in two ways: (1) fine-tune to condition on robot end-effector pose for 3D dynamics prediction on the PGND benchmark, and (2) fine-tune with an action head that predicts robot action chunks + 3D tracks, using 20 labeled demos + 100 actionless demo videos per task. Both post-training regimes outperform strong baselines on real robots; from-scratch ablations isolate the pretraining objective and dataset as the load-bearing contributions.

Key claims

3D point track completion (predict dense future 3D tracks of all observed points, given RGB-D + sparse partial tracks as context) is a rich pretraining objective for 3D dynamics that requires no robot action labels [§Abstract, Method fig].
A flexible Perceiver-IO visual encoder over DINOv2 features + a denoising transformer alternating point-level and global attention outperforms prior 3D neural-dynamics baselines (GBND, ParticleFormer, PGND, PTv3) on the same 2.9M-frame synthetic dataset across all 9 metric-object combinations, with MDE↓ 3.59 vs 9.01 on deformable, 2.40 vs 8.25 on articulated, 27.26 vs 61.97 on rigid (mean-10, cm) [Table — Dynamics prediction on synthetic held-out].
Flow matching and JiT beat direct regression as denoising objectives; with sufficient training and matched sampling protocols, JiT does not consistently outperform flow matching on this dataset [Table — Dynamics prediction on synthetic held-out].
Post-trained to condition on robot end-effector pose, PointZero-FT outperforms scene-specific GBND / PGND baselines on 4 of 6 real-world PGND benchmark scenes (best-of-10 oracle, cm): Bread 1.5 vs 2.0, Paperbag 1.6 vs 1.6, Cloth 4.0 vs 4.5, Box 2.4 vs 2.2, Rope 3.3 vs 3.9, Sloth 3.9 vs 4.3 [Table — PGND benchmark].
The pretraining objective is load-bearing: PointZero-Scratch (same architecture, no pretraining) collapses to 4.2 / 4.2 / 5.6 / 3.7 / 9.3 / 7.3 MDE across the six PGND scenes — 2–3× worse than the fine-tuned checkpoint on 4 of 6 [Table — PGND benchmark].
Post-trained with an action head (20 action-labeled demos + 100 actionless demonstration videos per task), PointZero-FT achieves the highest or joint-highest success rate on 6 of 7 simulated + real-world manipulation tasks, and beats DP3 on all seven — 99.8% Blockstack (sim, vs DP3 44.4 / ATM 40.0 / 3PoinTr 90.9), 93.1% Microwave (sim, vs 31.4 / 30.4 / 80.8), 100% Drawer (real, vs 70 / 30 / 90) [Table — Imitation-learning success rates].
The action head can also be trained with 20 labeled demos only (no actionless videos); pretraining still helps on the harder tasks — Blockstack 93.5→93.2, Microwave 81.4→91.9, Glass 66.6→79.5 with all-parameter training [Table 4], and larger gains when only the pretrained point-processing stream is frozen [Table 5].
Flow matching / JiT objectives learn a distribution over trajectories, not a single mean: different random seeds yield plausible but distinct future dynamics under identical input + conditioning tracks [§Diversity qualitative].

Method

Input: a single RGB-D frame plus $k$ sparse complete 3D point trajectories (context tracks). Output: dense future 3D trajectories for every observed scene point.

Architecture: a Perceiver-IO visual encoder ingests DINOv2 features from the RGB-D frame and produces a fixed set of latent tokens. A denoising transformer then alternates point-level attention (over the trajectory tokens) with global attention (over the shared latent scene set), letting per-point trajectory prediction stay coupled to scene-wide geometry. Because both the labels and the conditioning are just 3D point tracks, supervision is robot-free — any source of dense 3D tracks (synthetic simulation, or in principle any tracked real video) can produce training data. The paper compares three denoising heads: flow matching, JiT (Back to Basics: Let Denoising Generative Models Denoise), and direct regression; flow matching / JiT beat regression, and with matched sampling protocols the two denoising heads are close.

Two post-training regimes reuse the same pretrained trunk:

Action-conditioned dynamics. Fine-tune to accept robot end-effector pose as extra conditioning; predict future object dynamics only, evaluated on the PGND real-world benchmark.
Imitation learning. Add a lightweight action head that predicts robot action chunks jointly with the 3D tracks. Training uses 20 action-labeled demonstrations + 100 actionless demonstration videos per task. Two variants are reported: fine-tune all parameters, or freeze the pretrained point-processing stream and only train the new head + adapters.

Data: a purpose-built synthetic dataset of 2.9M frames with dense per-point trajectory annotations, spanning deformable / articulated / rigid objects with motions driven by randomized interactions. A held-out real-world evaluation set of 14 objects × 124 interactions is collected with trajectory labels supplied by Track4World: Feedforward World-Centric Dense 3D Tracking of All Pixels-adjacent tooling (FoundationStereo + CoTracker3). Dataset, checkpoints, and training recipe are released.

Results

Dynamics on synthetic held-out (cm, mean-10; lower better). PointZero-FM beats every baseline on every metric × object type:

Deformable: MDE 3.59 (next-best PTv3 9.20), CD 1.55 (3.91), EMD 3.18 (7.86).
Articulated: MDE 2.40 (PTv3 8.25), CD 1.77 (5.25), EMD 2.34 (8.00).
Rigid: MDE 27.26 (PTv3 61.97), CD 14.17 (29.87), EMD 25.41 (58.57).

Oracle-10 (best-of-10 with GT selection) drops errors further (2.80 / 1.96 / 19.91 MDE), consistent with the flow model sampling from a genuine multi-modal posterior [Table — synthetic held-out].

Action-conditioned dynamics on PGND (cm, best-of-10 oracle). PointZero-FT is best on 4/6 scenes (Bread 1.5 vs PGND 2.0; Paperbag 1.6 tie; Cloth 4.0 vs 4.5; Rope 3.3 vs 3.9; Sloth 3.9 vs 4.3), loses to PGND only on Box (2.4 vs 2.2). PointZero-Scratch is 2–3× worse than PointZero-FT on 4/6 scenes, isolating pretraining as the load-bearing lever [Table — PGND].

Imitation-learning success rates (%). PointZero-FT wins or ties on 6/7 tasks and beats DP3 on all seven:

Blockstack (sim): 99.8 vs 3PoinTr 90.9 / ATM 40.0 / DP3 44.4.
Microwave (sim): 93.1 vs 80.8 / 30.4 / 31.4.
Glass (sim): 95.9 vs 95.2 / 3.9 / 74.9.
Drawer (real): 100 vs 90 / 30 / 70.
Cup (real): 100 vs 100 / 30 / 80.
Paper (real): 70 vs 90 / 0 / 10 (only task where PointZero is not best).
Sock (real): 90 vs 90 / 0 / 20 [Table — IL success].

Scratch-vs-pretrained IL ablations (%, 1,000 rollouts / task, 20 labeled demos, no extra actionless videos). All-parameter training: Blockstack 93.5→93.2 (tie), Microwave 81.4→91.9 (+10.5), Glass 66.6→79.5 (+12.9), average 80.5→88.2 (+7.7). Freezing the pretrained point-processing stream: Blockstack 83.8→85.2, Microwave 89.5→90.1, Glass 49.1→64.6, average 74.1→80.0. The gain from pretraining is largest on the harder tasks and when the pretrained stream is protected from IL gradient noise [Tables 4–5].

Why it’s interesting

PointZero stakes out a distinct middle position in the world-foundation-models design space. The two well-defined poles on the World Foundation Models page are (a) pixel-rollout WFMs (Cosmos 3, Genie 3, LingBot-World, Runway GWM-1) that predict full frames and (b) latent-predictive JEPA-family WFMs (V-JEPA 2, VL-JEPA, LeWorldModel) that predict masked embeddings. PointZero picks a third interface — dense 3D point tracks — which is neither raw pixels nor learned latent embeddings. This is close in spirit to μ₀: A Scalable 3D Interaction-Trace World Model (µ₀’s 3D interaction traces feeding a frozen WM plus small action expert) but pushed further: point tracks are the supervision target itself, not just the interface, so pretraining and post-training use the same signal. It is also the natural WFM counterpart to Track4World: Feedforward World-Centric Dense 3D Tracking of All Pixels (world-centric dense 3D tracking as perception) — same underlying primitive, one on the perception side, one on the prediction side.

The action-agnostic pretraining plus small-data post-training result is a concrete positive datapoint for Sitzmann’s “generative-rollout as pre-training objective” thesis (The flavor of the bitter lesson for computer vision) at the 3D-track granularity — 20 action-labeled demos plus 100 actionless videos per task beats DP3 across the board and matches or exceeds a strong 3PoinTr baseline on 6/7 tasks. Complements LUCID: Learning Embodiment-Agnostic Intent Models from Unstructured Human Videos for Scalable Dexterous Robot Skill Acquisition LUCID on the Human-to-Robot Retargeting concept axis: LUCID retargets task intent (short-horizon 3D object flow + palm-pose reference), while PointZero retargets dense 3D scene dynamics through the same action-free-pretrain-then-add-action-head recipe. Contrasts with Inference-time Physics Alignment of Video Generative Models with Latent World Models (VJEPA-2 as a frozen physics-plausibility reward) — same JEPA-adjacent latent-predictive lineage, but PointZero exposes an explicit geometric prediction target that downstream policies can consume directly rather than reading out a scalar surprise signal.