π*0.6: a VLA That Learns From Experience (RECAP)

TL;DR

Physical Intelligence introduces π*0.6, a VLA trained with RECAP (RL with Experience and Corrections via Advantage-conditioned Policies), an iterated-offline-RL recipe that lets a flow-matching VLA improve from heterogeneous data — demonstrations, autonomous on-policy rollouts with sparse outcome rewards, and human-gated DAgger interventions. The key trick is advantage conditioning: a multi-task distributional value function estimates per-action advantages on the collected data, those advantages are binarized into a “positive/negative” text token in the VLA’s prefix, and the policy is trained with plain supervised learning on the full dataset — sidestepping the log-likelihood and PPO-style scaling problems that have blocked end-to-end RL fine-tuning of large flow-matching VLAs. The result is a generalist policy that, after a few rounds of on-robot RL, can fold laundry in real homes, assemble cardboard boxes for factory packaging, and pull espresso drinks on a professional machine for 13 straight hours — doubling throughput and halving failure rates on the hardest tasks vs. the SFT baseline.

Key claims

RECAP combines three data sources — demonstrations, autonomous on-policy rollouts with sparse outcome rewards, and expert teleoperated interventions (human-gated DAgger) — in a single iterated-offline-RL loop, and pre-trains the VLA with offline RL before any on-robot collection [§I, §IV].
Policy extraction is done via advantage conditioning (a CFGRL-style classifier-free-guidance variant): the VLA is trained on all data with supervised learning, conditioned on a binarized “Advantage: positive/negative” text token derived from a separately-trained value function — avoiding policy gradients entirely [§IV-B, Eq. 3].
The value function is a multi-task distributional value head sharing the VLA architecture but with a smaller VLM backbone, trained by cross-entropy over B=201 discretized return bins on Monte-Carlo returns of the behavior-policy data [§IV-A, Eq. 1].
Human-corrected actions during autonomous rollouts are force-labeled It = True (positive advantage) regardless of value-function output, encoding the assumption that experts always provide good corrective actions [§IV-B].
π*0.6 is built on π0.6 (a successor to π0.5 with a Gemma 3 4B VLM backbone, 860M action-expert, and the Knowledge Insulation training recipe), adding only the binarized-advantage text input — the action expert is unchanged [§V-A, Fig. 3].
Discrete tokens (FAST-tokenized actions and high-level sub-task text ℓ̂) and continuous flow-matching actions are predicted independently in a single forward pass, with the advantage token placed after ℓ̂ but before the actions so only action log-likelihoods are affected [§V-B].
On the hardest tasks RECAP more than doubles task throughput and roughly halves the task failure rate vs. the imitation-only baseline [§I, Abstract].
Practically useful robustness numbers reported: 13 hours straight of espresso drinks on a professional machine, >2 hours of folding novel laundry in a new home, and box assembly used for real factory packaging [§I].
The advantage threshold ε_ℓ (per-task) replaces the CFG-style attenuation β used in prior CFGRL work — the authors argue high CFG weights drive the action distribution to the corners of its support and don’t affect autoregressive tokens, so a threshold on the conditioning indicator is preferred [§IV-B, footnote 2].
Advantage conditioning is shown to significantly outperform policy-gradient-style policy extraction in their setting — a critical claim because policy gradients on flow-matching VLAs require tractable log-likelihoods that the flow model doesn’t readily provide [§II, §VI].
The KI (Knowledge Insulation) recipe is preserved: the flow-matching action expert is trained with a stop-gradient that prevents action losses from affecting the VLM backbone, while the VLM still trains on next-token prediction over discretized actions, sub-tasks, and web vision-language co-training data [§V-A].

Method

RECAP iterates three subroutines: (1) collect autonomous rollouts on each task, label each episode with a sparse outcome reward, optionally inject human-gated DAgger interventions; (2) train a multi-task language-conditioned distributional value function p_φ(V | o_t, ℓ) ∈ Δ^B by cross-entropy against discretized Monte-Carlo returns over all data collected so far; (3) train the VLA on all the data with supervised log-likelihood, conditioning on a binarized “Advantage: positive/negative” text token where I_t = 1(A^{π_ref}(o_t, a_t, ℓ) > ε_ℓ). Pre-training runs steps (2) and (3) on a multi-task, multi-robot demonstration corpus; specialists then run all three steps for one or more iterations per task.

The VLA itself is π0.6: Gemma 3 4B base VLM, 860M flow-matching action expert with stop-gradient on the VLM, trained under the Knowledge Insulation recipe with mixed next-token-prediction (sub-task ℓ̂, FAST-tokenized actions a_ℓ, web VL data) and flow-matching action loss. π*0.6 adds the binarized-advantage text token after ℓ̂ so it modulates only the action heads. Inference: sub-task prediction at low frequency, action chunks at 50 Hz, advantage token fixed to “positive” at deployment.

The closed-form justification (§III, §IV-B) starts from the KL-regularized RL solution π̂(a|o) ∝ π_ref(a|o) · exp(A^{π_ref}(o,a)/β), rewrites it via Bayes as π̂(a,|o,ℓ) ∝ π_ref(a|o,ℓ) · (π_ref(a|I,o,ℓ)/π_ref(a|o,ℓ))^β, and notes that for β=1 the improved policy is simply the conditional π_ref(a|I,o,ℓ) — which is exactly what the supervised conditional objective learns when trained jointly on (o,ℓ) → a and (o,ℓ,I) → a. This is the CFGRL trick.

Results

The paper’s headline quantitative claim is “more than doubles task throughput and roughly halves the task failure rate” on the hardest tasks vs. an imitation-only baseline [§I, Abstract]. Qualitative deployment numbers: 13-hour continuous espresso-drink session on a professional machine, 2+ hour novel-laundry folding in a new home, factory-grade box assembly. Detailed per-task tables, ablations of advantage-conditioning vs. policy-gradient policy extraction, and the value-function visualization (Fig. 4 — VF correctly drops on a knocked-over water filter and rises on successful fridge-open) are in the latter half of the paper (truncated at fetch time at 30KB).

Why it’s interesting

This is the first filed example of a flow-matching VLA trained end-to-end with offline RL on real-robot data, and it lands in a spot the wiki’s existing reasoning-RL story doesn’t quite cover. Reasoning RL tracks GRPO-and-cousins as the default post-training primitive — but every filed instance (text agents, VLM-gym, Video-Thinker, HuDA) leans on tractable per-token log-likelihoods, which a flow-matching action expert doesn’t have. RECAP’s answer is to skip the gradient entirely: convert the value function’s output into a discrete text token in the prefix and let the VLA’s existing supervised loss do the work. That’s the same CFGRL primitive the wiki tracks elsewhere (Classifier-Free Guidance variants), now applied to policy extraction.

This also extends the World Foundation Models / VLA stack story: where Gemini Robotics 1.5 brings AI agents into the physical world keeps the policy-training details closed and NVIDIA Unveils New Open Models, Data and Tools to Advance AI Across Every Industry (Isaac GR00T N1.6 / Cosmos Reason 2) is the open-stack counterpart, π*0.6 is the open-recipe RL-finetuning side — answering “how do you actually improve a generalist VLA with autonomous experience?” with a concrete, ablated method. Contrasts with Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrations, which sidesteps the policy problem by reading 6D pose out of generated videos; RECAP is the opposite bet — train the policy hard, use sparse outcome rewards plus human-gated interventions to drive improvement.

The “interventions force I=True” trick is also a clean way to integrate DAgger-style corrections into an RL loop without a separate imitation/RL switch — worth contrasting with the streaming-rollout MiroThinker recipe (MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling) where corrections enter as filtered trajectories rather than label-overridden actions.