Reconstruct, Practice, Go Real: Guided Self-Improvement for Embodied Agents

TL;DR

RPG (Berkeley, Abbeel/Haghtalab/Sastry labs; Wang, Jiang, Deng, Xue, Ye, Duan, Qi) is a no-weight-update self-improvement framework for embodied agents: identify manipulation capabilities from an offline dataset, construct related practice tasks in simulation, and let a multimodal LLM diagnose per-rollout failures using execution feedback + privileged simulator state + dataset videos, then edit a persistent system prompt + reusable symbolic skill library. Candidate changes face cross-task evaluation before being retained for reuse. On held-out initializations of 22 manipulation tasks, RPG improves success from 28.6% after the first practice round to 95.0% after 15 rounds, outperforming ASPIRE (75.5%) and CaP-Agent0 powered by GPT-6 Astra Pro (60.0%). After a common calibration and hardware-adaptation procedure, the frozen system succeeds in all 30 physical trials (10 trials × 3 tasks). The shipped artifact is a prompt + code library rather than policy weights — same class of system as ASPIRE: Agentic /Skills Discovery for Robotics ASPIRE and RoboRSI: Stable, Efficient, and Reusable Robot Self-Evolution in Complex Real-World Environments RoboRSI but with explicit cross-task validation gates on candidate changes and a sharper measured sim-success → zero-shot-real-success transfer.

Key claims

• Capability identification from offline data is the entry point: the framework mines manipulation capabilities present in an offline dataset and constructs related practice tasks in simulation that exercise those capabilities, rather than practicing on the target task directly [§Abstract].
• The improvement loop combines three feedback channels — execution feedback, privileged simulator state, and available dataset videos — into one diagnostic context for a multimodal LLM, so failure analysis uses more than the rollout-level pass/fail signal that per-primitive-trace systems like ASPIRE already criticized [§Abstract].
• The system edits three distinct persistent artifacts: new reusable symbolic skills, refined existing skills, and a revised system prompt — making the policy a prompt + code library rather than a model with trained weights [§Abstract].
• Cross-task evaluation tests individual candidate changes and merged revisions before they are retained for reuse — explicit validation gate on candidate skills, structurally similar to Zetta’s shadow-replay gates and ACM’s validated compaction discipline [§Abstract].
• The frozen deployed system uses a multimodal LLM to coordinate perception and robot control at test time against the system prompt + skill library produced by the loop [§Abstract].
• Success on 22 held-out initializations rises 28.6% → 95.0% over 15 practice rounds [§Abstract].
• RPG outperforms baseline agentic systems on held-out initializations by a large margin: 95.0% vs ASPIRE 75.5% vs CaP-Agent0 (GPT-6 Astra Pro) 60.0% [§Abstract].
• After a common calibration + hardware-adaptation procedure, the frozen system succeeds in 30/30 physical trials across three tasks (10 trials each) [§Abstract].
• Weight updates are not required for the self-improvement to compound — the shipped artifact is prompt + code [§Abstract].

Method

(Only the abstract-level mechanism is retrievable at filing time; full §-level architecture not readable from the abs page.) RPG is organized as a three-stage loop. Reconstruct: given an offline dataset of robot manipulation, the system extracts the set of manipulation capabilities present in it and reconstructs related practice tasks in a simulator. These practice tasks are not the target evaluation task — they are a curriculum of capability-related tasks derived from the offline data. Practice: during practice, a multimodal LLM runs rollouts in simulation and ingests three distinct feedback channels — the executor’s own execution feedback (status codes, errors), privileged simulator state (ground-truth poses, contact events, etc. unavailable at real deployment), and offline dataset videos as reference — then diagnoses failures. Diagnosis produces three kinds of candidate edits: (a) new symbolic skills, written as compact reusable programs; (b) refinements to existing skills; (c) revisions to the system prompt. Candidate edits run through a cross-task evaluation stage: both individual changes and merged revisions are tested across the practice-task set, and only those that improve or preserve performance across the broader set are retained. The persistent skill library + system prompt grow across rounds. Go Real: at deployment, the multimodal LLM is frozen; it reads the final system prompt + skill library and coordinates perception + robot control on real hardware after a common calibration/hardware-adaptation procedure. No weight updates occur at any point in the pipeline.

Results

Headline numbers from the abstract (§-level results tables not retrievable at filing):

• 22 held-out manipulation tasks (held-out initializations, in sim):
  • RPG round 1: 28.6%
  • RPG round 15: 95.0%
  • ASPIRE: 75.5%
  • CaP-Agent0 (GPT-6 Astra Pro backbone): 60.0%
• Real-robot deployment (3 tasks × 10 trials = 30 trials) after common calibration + hardware-adaptation: 30/30 success (100%) with the frozen system (same system prompt + skill library produced by the sim loop).

The 15-round improvement curve from 28.6% to 95.0% is the clearest sim datapoint for monotone compounding in the self-evolving-skill-library family on a held-out eval of ≥20 tasks. The 30/30 real-robot result stands out against the small-N real-robot results typical of this family (ASPIRE’s own real-world cross-embodiment transfer reports token-cost reductions on 20-trial per-task budgets rather than full-success real-world runs).

Why it’s interesting

RPG is the sharpest head-to-head datapoint yet on the “self-evolving skill library beats frontier-LLM one-shot code-as-policy” question that ASPIRE: Agentic /Skills Discovery for Robotics ASPIRE opened. ASPIRE reported +77pp on LIBERO-Pro Object and 4%→31% on held-out LIBERO-Pro Long; RPG reports its own 22-task held-out suite at 95.0% RPG vs 75.5% ASPIRE vs 60.0% CaP-Agent0, a direct within-paper comparison that places RPG above ASPIRE on the same evaluation axis. The method differs from ASPIRE on three specific moves: (1) the practice tasks are reconstructed from an offline dataset’s capability inventory rather than being handed to the agent, extending the capability-mining idea into the practice-task curriculum itself; (2) the diagnostic context combines execution feedback + privileged simulator state + dataset videos (ASPIRE uses per-primitive multimodal traces but not reference videos from a prior dataset); (3) RPG makes the cross-task validation gate explicit on both individual and merged revisions, which is structurally similar to Zetta ζ: An Efficient Closed-Loop Embodied Harness for Self-Evolving Physical Intelligence Zetta’s five-stage shadow-replay + paired-seed + held-out-seed promotion gate — but applied to a per-skill artifact rather than runtime critic-recovery pairs. Sibling in framing to RoboRSI: Stable, Efficient, and Reusable Robot Self-Evolution in Complex Real-World Environments RoboRSI (four-agent harness over a four-level skill tree with real-environment validation) and to Embodied Turing Machines: Stateful Code for Robot Recursive Self-Improvement COAP’s “shared code library is the policy” position, but RPG retains the frontier-LLM-at-test-time orchestrator while COAP removes it. One caveat: the abstract reports results against ASPIRE and CaP-Agent0 but does not disclose the backend LLM used by RPG for the loop (CaP-Agent0 is explicitly powered by GPT-6 Astra Pro); until the full paper is readable, the frontier-LLM-cost picture that Astranomy: An Empirical Study of GPT-6 Astra as a Native Robot Policy Astranomy quantified for related agentic systems (~$36/successful episode on Astra) cannot be directly compared.

See also

• ASPIRE: Agentic /Skills Discovery for Robotics — ASPIRE: the closest sibling in the self-evolving-skill-library family; RPG beats it 95.0% vs 75.5% on its own 22-task held-out eval
• ASPIRE — self-evolving sensorimotor skill library that compounds across robots and the sim-to-real gap (Jim Fan / NVIDIA GEAR) — Jim Fan’s June 2026 ASPIRE announcement, the original “ship a code library, not weights” framing that RPG inherits
• Playful Agentic Robot Learning — RATs: play-stage skill library acquired without downstream tasks; RPG’s “reconstruct capabilities from offline data” is a different curriculum-generation answer to the same problem
• CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation — CaP-X: the Code-as-Policy framework that CaP-Agent0 (RPG’s baseline) is instantiated against
• RoboRSI: Stable, Efficient, and Reusable Robot Self-Evolution in Complex Real-World Environments — RoboRSI: four-agent harness with live-environment validation; RPG’s cross-task validation gate shares the “revisions must survive broader evaluation” discipline
• Zetta ζ: An Efficient Closed-Loop Embodied Harness for Self-Evolving Physical Intelligence — Zetta’s shadow-replay + paired-seed + held-out-seed gate is structurally close to RPG’s “both individual and merged revisions tested across tasks” validation
• Embodied Turing Machines: Stateful Code for Robot Recursive Self-Improvement — COAP: the “no LLM at test time, library is the policy” extreme; RPG retains the test-time orchestrator while sharing the “capability lives in authored code” position
• AI-for-AI Research — RPG is a per-skill-library instance of the broader cluster where agents mine their own history to compound improvement
• VLA Models — adds another row to the recipe-lever board distinct from end-to-end VLAs: no learned action head, no policy weights, just a growing symbolic skill library coordinated by a frontier LLM at test time
• Tool-Use Agents — the test-time orchestrator reads the system prompt + skill library as its tool interface
• Simulator-in-the-Loop Refinement — adjacent on the “practice in sim before real deployment” axis, but RPG is training-time self-improvement over a persistent library, not per-episode inference-time refinement
• RL Environment Platforms — the practice-task generation + sim rollout substrate the loop runs on