μ₀: A Scalable 3D Interaction-Trace World Model

TL;DR

µ₀ is a world model whose interface is neither dense pixels nor embodiment-specific actions but smooth 3D trajectories of salient interaction points — objects, tools, hands, and contact regions — represented as B-spline control points predicted by a modular trace expert on top of a pretrained vision-language backbone. To train at scale without action labels, the authors build TraceExtract: an automatic pipeline that selects keypoints from any video, constructs globally aligned 3D traces, and pairs them with hierarchical language captions. The frozen µ₀ then plugs into downstream embodiments via action experts and reportedly matches action-supervised VLA baselines like π₀ on robot manipulation, despite never seeing an action label during pretraining. The bet is that 3D traces are a compact, embodiment-agnostic motion interface that ducks both the pixel-reconstruction tax of video world models and the embodiment-coupling tax of direct action models.

Key claims

3D traces over salient interaction points (objects, tools, hands, contacts) are a compact, embodiment-agnostic motion interface that avoids the dense-appearance reconstruction tax of pixel-space video world models and the embodiment-label tax of direct action models [Abstract].
The TraceExtract supervision pipeline produces 3D trace + hierarchical caption annotations automatically from diverse video sources by keypoint selection, global trace alignment, and motion-segment captioning — i.e. enables action-free pretraining of a world model from heterogeneous web/embodiment video without per-embodiment action labels [Abstract].
µ₀ represents each query via B-spline control points and predicts future traces with a modular trace expert head attached to a pretrained vision-language backbone [Abstract].
µ₀ outperforms baseline 2D and 3D trace prediction methods, including dedicated trace-prediction models and tokenized-VLM approaches [Abstract / Experiments].
Because µ₀ is frozen and reusable, downstream embodiments are integrated by pairing it with embodiment-specific action experts rather than retraining the world model [Abstract].
Despite action-free pretraining, trace-conditioned policies on top of µ₀ achieve performance competitive with VLA models pretrained with action supervision (π₀ named explicitly) [Abstract].

Method

µ₀ inverts the prevailing two design choices for embodied world models. Pixel-space video models (e.g. Cosmos Predict, Veo, Genie 3) spend most of their model capacity reconstructing appearance, only a fraction of which is causally relevant to manipulation. Direct action models (VLAs like π₀ or OpenVLA) instead model the action distribution directly, but require embodiment-specific action labels, which fragments the data pool by hardware. µ₀’s alternative is to predict 3D trajectories of a small set of salient interaction points — objects, tools, hands, contact regions — over a future horizon. This is dense enough to specify what should happen mechanically, sparse enough to discard appearance, and structurally identical across embodiments.

The architecture is a pretrained vision-language backbone with a modular trace expert head. For each query interaction point, the head emits B-spline control points that parameterize a smooth 3D trajectory; B-splines act as a low-frequency prior over trajectories, replacing per-timestep token generation. To enable training from heterogeneous video at scale, the authors introduce TraceExtract: keypoints are selected on interaction-relevant scene elements, trajectories are constructed and globally aligned across frames, and motion segments are paired with hierarchical language captions. This yields (video, hierarchical caption, future 3D traces) tuples that supervise µ₀ without any action labels.

At deployment, µ₀ is frozen. To control a specific robot, an action expert is trained that consumes µ₀’s predicted 3D traces (and the observation) and emits embodiment-specific actions. This factors the cross-embodiment generalist problem into “shared trace-prediction prior” + “embodiment-local action decoder,” analogous to how D-MPC factorizes a diffusion dynamics model from an action proposal model (Diffusion Model Predictive Control).

Results

The paper reports two top-line results in the abstract:

µ₀ beats baselines on both 2D and 3D trace prediction, including dedicated trace-prediction models and tokenized-VLM methods that fold actions into text tokens [Abstract].
Trace-conditioned policies built on top of frozen µ₀ + an embodiment-specific action expert are competitive with VLA models pretrained with action supervision, with π₀ called out by name [Abstract].

The abstract does not disclose specific benchmark suites, success rates, or ablations of TraceExtract’s automatic-supervision pipeline; those would need to be read off the PDF body and figures, which were not retrieved at filing time. The headline framing positions µ₀ against π₀ (π*0.6: a VLA That Learns From Experience (RECAP)) as the action-supervised reference and tokenized-VLM/trace-prediction baselines as the trace-prediction reference.

Why it’s interesting

µ₀ is the most direct articulation on the wiki so far of a third path between two design poles the cluster has been debating. Pixel-space generative-rollout world models — Genie 3, Cosmos Predict, NewtonGen, PAN — are the dominant generative answer, and Sitzmann’s “bitter lesson of CV” essay (The flavor of the bitter lesson for computer vision) is their normative defense: video pretraining as the perception substrate, no explicit 3D in the model. VLAs at the other pole — π₀.7 (π0.7: A Steerable Robotic Foundation Model with Emergent Compositional Generalization), Spirit-v1.5 (Spirit-v1.5: Clean Data Is the Enemy of Great Robot Foundation Models), Embodied-R1.5-VLA (Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models) — encode actions directly. µ₀ picks neither: it predicts 3D traces as an intermediate language. That this is reportedly competitive with π₀ on downstream policies, while being trained without action labels, is the specific claim that would, if it survives scrutiny, validate the trace-as-interface bet against both ends.

It is a clear lineage with DynaFLIP: Rethinking Robotics Perception via Tri-Modal-Dynamics Guided Representation — the immediately preceding paper from the same Seungjae Lee / Jusuk Lee / Jia-Bin Huang / Furong Huang group. DynaFLIP’s argument was that the vision encoder should bake in dynamics via image-transition × language × 3D-flow triplets, replacing static-image backbones (DINOv2, SigLIP) under the policy. µ₀ is the same group’s next move at a higher level of abstraction: instead of pushing dynamics into the encoder, push the world model’s interface to 3D trajectories of interaction points, and let the policy be a thin action expert on top. The two papers also share the TraceExtract-style “extract 3D supervision from web video” recipe — DynaFLIP’s 260K-trajectory pipeline is plausibly the data ancestor of TraceExtract.

It also gives the wiki’s debated Jia-Bin Huang quote-RT — new dynamics-aware vision encoder for robotics (@jusukle) / Jia-Bin Huang quote-RT — new dynamics-aware vision encoder for robotics (@jusukle) teaser a second anchor: that quote-RT was about DynaFLIP’s encoder, and µ₀ is the follow-up world model from the same authors. Sibling-paper context that should be appended once a curator confirms.

Worth contrasting with Direct Video-Action Models — Causal Video Models Are Data-Efficient Robot Policy Learners (Direct Video-Action Models) — DVA uses a video model as the policy with a small inverse-dynamics decoder; µ₀ replaces the video model with a trace model. Both share the architectural commitment that the policy is a small decoder on top of a heavy embodiment-agnostic pretraining target, but disagree on what that target should emit (dense pixels vs. sparse 3D traces).