τ₀-VLA: A Hierarchical Robot Foundation Model with World-Model-Guided Test-Time Computation

TL;DR

τ₀-VLA is a hierarchical robot foundation model that reframes high-level subtask generation as a compute-scalable inference problem: at each step, a memory-augmented VLM proposes a subtask, and when token confidences flag uncertainty, a world-model-guided beam search expands candidate subtask sequences, predicts each candidate’s terminal head-camera image with a video world model, scores those imagined outcomes with a value model, and lets a reflective model commit to the final subtask. The low-level policy is a Mixture-of-Transformers action expert with conditional flow-matching over a unified 40-dimensional state/action space trained on 40,115 hours of heterogeneous real-world data. Reported to sustain long-horizon real-world episodes up to ~12 minutes (room cleaning, cooking, tea-making, laundry collection) across multiple embodiments, with next-subtask accuracy climbing substantially as inference budget grows.

Key claims

The high-level policy makes subtask selection an inference-time-scalable operation via a propose-predict-evaluate beam search: a VLM proposes N candidates per retained branch, a world model predicts each candidate’s terminal head-camera image, a value model scores the predicted outcome, and a reflective model conditions on retained branches to emit the final subtask — inference budget is scaled by branching factor N, beam width B, and depth D [§IV-B, Alg. 1].
Visual outcome prediction occurs before commitment, not after — this is contrasted explicitly with π0.7, where the world model generates a subgoal image for a subtask already produced by the high-level policy [§II-B].
Correctable execution memory is trained by perturbing memories derived from existing demonstrations and requiring the policy to repair records that lag behind, run ahead of, or misrepresent progress — no additional annotation needed [§I, §IV-B].
A confidence-based adaptive router (token-confidence statistics from the proposal model’s forward pass) decides whether to commit to the direct proposal or trigger the search; the routing rule uses task-specific thresholds calibrated on held-out data [§IV-B, App. C-J].
The low-level policy uses a unified 40-dimensional state/action space with per-sample masks over end-effectors/arms/grippers/waist/mobile-base, letting one model span fixed-base manipulation, bimanual coordination, and mobile whole-body control without embodiment-specific heads [§IV-C].
Masked conditional flow matching supervises only active channels of the unified action space and projects inactive channels before every velocity-field evaluation [§IV-C, Eq. 14].
More inference-time compute yields higher next-subtask prediction accuracy — on out-of-distribution Book Organization, accuracy rises from 50% to 74%, and this translates into higher closed-loop success (per the announcement thread; specific in-paper numbers in appendix tables not shown in retrieved excerpt).
The reflective model’s output is not restricted to the retained candidate set — it may generate a subtask that does not appear in any retained branch [§IV-B].

Method

At each logical step t, the proposal model P consumes the task instruction, execution memory, prior subtask, and current observation to produce both an updated memory and a direct proposal z_dir_t. From the same forward pass’ token-confidence statistics, an adaptive router chooses a fast route (commit to z_dir_t) or a Test-Time Computation (TTC) route. On the TTC route, beam search independently samples N candidate subtasks from P for each retained branch; for each candidate, the world model W predicts the terminal head-camera image starting from the branch’s most recent imagined image, and the value model V scores that predicted image against the global task instruction. Branches are pruned globally to top-B by cumulative score and expanded to depth D. The reflective model F then generates the final subtask z_t* conditioned on the retained branch summaries plus the observation-aligned real context — free to deviate from the candidate set. The low-level policy π_θ pairs a Qwen3.5-2B VLM backbone with a Mixture-of-Transformers action expert; joint attention connects the two streams at every full-attention layer, and the action expert integrates a conditional flow-matching velocity field from Gaussian noise to a length-H action chunk (with active-channel masking throughout).

Data: 40,115 hours combining human teleop demonstrations, autonomous policy rollouts, and UMI-style hand-held recordings across fixed-base / mobile / bimanual embodiments, interleaved with multimodal VQA / grounding / spatial-reasoning data to preserve the VLM backbone’s semantic capabilities during action learning.

Results

Long-horizon real-world evaluation across multiple robot embodiments on room cleaning, meal preparation, tea making, and laundry collection with episodes lasting up to 12 minutes [§I, §V].
Hierarchical TTC substantially improves closed-loop task success over whole-task inference using the same low-level policy [§I].
Next-subtask prediction accuracy improves with inference-budget scaling — quoted 50% → 74% on OOD Book Organization from the announcement thread; per-task closed-loop numbers deferred to the paper’s evaluation section (partially truncated in retrieved excerpt).
Ablation analyses of execution memory, consequence-aware search, and the compute-accuracy trade-off are provided [§I].

Why it’s interesting

Adds a distinct row to the VLA Models recipe board: none of the prior filed VLA recipes (RECAP action-RL, clean-teleop, unified-VLM pointing, frozen-WFM+action-expert, native video+action pretraining) use world-model-guided beam search over language subtasks as an inference-time-scalable interface between the high-level VLM and the low-level policy. It contrasts most sharply with π0.7: A Steerable Robotic Foundation Model with Emergent Compositional Generalization (π0.7), where the world model images a subgoal after the high-level policy has already committed — τ₀-VLA moves visual prediction to before commitment and uses it to compare candidates. It also directly extends the “search over discrete manipulation plans with a learned critic” pattern of VINE: Taming Generative Control Policies for Reinforcement Learning into the hierarchical VLA regime and generalizes the DriftWorld: Fast World Modeling through Drifting “WFM as inference-time planner” pattern from action-space rollouts to open-ended language-subtask rollouts.

For the Inference-Time Scaling cluster, this is the first filed robotics analog of the RLM / MiroThinker / PARL scaling axes: extra inference compute is spent on world-model-scored beam search over subtask sequences rather than on longer chains of thought or parallel LM dispatch. The confidence-based router adds an adaptive-budget primitive similar to Foveated Reasoning: Stateful, Action-based Visual Focusing for Vision-Language Models‘s foveation trigger, but applied to hierarchical subtask decisions instead of visual tokens.