ASPIRE: Agentic /Skills Discovery for Robotics

TL;DR

ASPIRE (NVIDIA GEAR + UMich SymbioticLab + UIUC + UC Berkeley AUTOLab + CMU LeCAR, Lu et al.) is a continual-learning robotic system in which coding agents refine robot control programs under code-as-policy and consolidate validated repairs into a persistent skill library — “training” is skill refinement, not gradient descent, and the shipped artifact is a code repo of sensorimotor skills instead of weights. Three components run in a closed loop: (1) a robot execution engine that exposes per-primitive multimodal traces (RGB keyframes, grasp candidates, planner return codes, overlays) so the agent can autonomously diagnose failures rather than seeing only rollout-level pass/fail; (2) a growing skill library of validated-repair patterns (failure signature + when-to-apply condition + repair strategy + code sketch); (3) an evolutionary search over control-program populations that broadens exploration beyond local repair loops. Against CaP-Agent0 (CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation) and VLA baselines (OpenVLA, π0), the paper reports up to +77pp on LIBERO-Pro perturbation suites, +72pp on Robosuite bimanual handover (20→92%), +32pp on BEHAVIOR-1K long-horizon tasks, and 31% zero-shot success on held-out LIBERO-Pro Long vs 4% for prior methods. V1.0 was open-sourced on 2026-10-16 at github.com/nvlabs/aspire with reported 99–100% success across LIBERO-Pro / Robosuite / BEHAVIOR-1K when the backend model is Claude Opus 5.5 or GPT-6 Astra.

Key claims

The robot execution engine replaces coarse rollout-level feedback with per-primitive multimodal traces (invoked API, inputs, outputs, return status, RGB keyframes before/after the call, overlays, grasp candidates, planner return codes) so the agent can selectively inspect salient logs rather than reasoning over full video [§2.1].
The skill library stores heterogeneous repair knowledge — localization heuristics, perception prompts, grasping constraints, navigation recovery strategies, motion primitives, scene-understanding routines, debugging workflows — induced bottom-up from validated repairs rather than pre-specified as a taxonomy [§2.2].
Each skill is stored as compact in-context guidance: failure signature + when-to-apply condition + repair strategy + optional code sketch, and only the coordinator-audited reusable patterns are admitted into the shared library [§2.2].
Evolutionary search over control-program populations is the inner-loop exploration mechanism: in each round the agent proposes a population of candidate programs conditioned on top-performing prior programs and their remaining failure traces, with the next round conditioned on both successes and residual failure modes [§2.3, Algorithm 1].
A coordinator–actor architecture avoids context-window and distribution-mixing issues: the central coordinator manages the shared skill library and dispatches actor coding agents; actors do not exchange chat histories or raw rollouts, only distilled skills [§2].
On LIBERO-Pro, Aspire improves over the strongest baseline by +77pp (Object), +41.5pp (Goal), and +42.5pp (Spatial) averaged across position and task perturbation axes [§3.4, Fig. 4].
On Robosuite, Aspire preserves near-saturated performance on easier contact-rich tasks and lifts bimanual handover from 20% → 92% [§3.4].
On BEHAVIOR-1K navigate-and-pick-up-radio, task-level success rises from 56% (CaP-Agent0) → 88% (Aspire); Aspire also outperforms human-expert-written programs on navigation and task success on BEHAVIOR-1K [§3.4].
The skill library enables zero-shot transfer: with the library accumulated on LIBERO-90, Aspire reaches 31% success on held-out LIBERO-Pro Long tasks while CaP-Agent0 saturates at 4% despite its reliance on test-time reasoning and retries [§3.5, Fig. 5].
Success scales with library size: Fig. 5(b) shows monotone improvement in held-out LIBERO-Pro Long success as the skill library grows from empty to full [§3.5].
Simulation-discovered skills transfer across embodiments to a real bimanual YAM robot (different perception, calibration, control stack, and APIs from the Franka simulator), reducing debugging tokens by nearly an order of magnitude on soda-can lifting (61.94M → 6.58M total tokens, 13/20 → 19/20 success) and enabling drawer manipulation (0/20 → 11/20) that the no-skill baseline cannot solve within its larger token budget [§3.6, Table 1].
Ablation decomposes the two load-bearing components: the robot execution engine alone raises LIBERO-Pro macro-average success from 14% → 62%, and adding evolutionary search further lifts it to 72% [§3.7, Fig. 6a/b].
Evolutionary-search budget shows diminishing returns: success on low-performing tasks improves steadily in the first few rounds, then more gradually, suggesting multi-hypothesis sampling quickly recovers alternatives missed by single-iteration debugging [§3.7, Fig. 6c].
Named limitations: no fully autonomous real-world lifelong loop yet (reset / safety / success detection still manual); the agent depends on a frozen frontier LLM (Claude Opus 4.6 with a 1M-token context) and smaller/weaker LLMs have not been shown to sustain the loop; the agent writes programs against a pre-defined perception/planning/control API and cannot propose new primitives; the library needs retrieval/pruning/ranking/re-validation mechanisms to scale; and the debug + evolutionary-search loop is compute-intensive (many LLM calls and simulator rollouts per task) [§5].

Method

Aspire adopts a coordinator–actor architecture over three components [Fig. 1]. The robot execution engine wraps each perception, planning, grasping, and control call to record the invoked API, inputs, outputs, return status, and multimodal evidence (RGB keyframes immediately before and after the call, overlays, grasp candidates, object poses, motion-planning results); the agent sees evidence localized to the implicated primitive rather than full video frames. A worked BEHAVIOR-1K example [Fig. 2] shows this attribution flow: in a navigate-and-pick-up-radio episode, perception correctly returns a radio pose but repeated navigate_to_pose calls return PLANNING_ERROR; the agent inspects return values + logs, infers the sampled navigation target lies within the ~20cm collision buffer of the table edge, and writes a multi-angle approach routine that samples alternative directions around the radio. The skill library stores validated repairs as compact in-context guidance (failure signature + when-to-apply condition + repair strategy + optional code sketch); the admitted skill for the radio case is a navigation recovery pattern — “when planner errors recur near an obstacle boundary because sampled target poses fall inside the collision buffer, sample alternative approach directions around the object before retrying” — not a complete radio-pickup program. Actors report structured findings (failure mode + validated fix + transferable pattern); the coordinator audits these for API-policy compliance and only promotes repairs that passed debug validation. Evolutionary search [Algorithm 1] takes a population of candidate programs each round, conditions the next round on both the top performers and their remaining failure modes, and admits generalizing repair patterns into the library after search concludes.

Experimentally the setup uses Claude Opus 4.6 with a 1M-token context as the sim coding agent, writing Python robot programs against CaP-X (an open-source code-as-policy framework built on MuJoCo Playground; CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation); real-robot runs use OpenAI Codex (GPT-5.5, reasoning-xhigh) on a bimanual YAM station [§3.1]. Debug/evaluation seeds are disjoint; Aspire learns on a small debug split and reports success on larger held-out eval seeds with one generated program per LIBERO-Pro/Robosuite task, while CaP-Agent0 regenerates per seed with per-episode test-time reasoning and retries [§3.3]. The V1.0 open-source release (2026-10-16) also reports a Rubik’s Cube solve on real hardware driven by GPT-6 Astra, and a success-vs-API-cost sweep across twelve coding-agent models from GPT-6 Luna (15% on 10 LIBERO-Pro tasks) through Claude Opus 5.5 (99%) — strongest models are not the most expensive, and Sonnet 5.5 sits on the cost-success frontier at ~$12/task.

Results

Headline numbers from the paper [Fig. 4, Fig. 5, Table 1]:

LIBERO-Pro perturbation suites: averaging over Pos and Task perturbation axes, Aspire gains +77pp on Object, +41.5pp on Goal, +42.5pp on Spatial over the strongest baseline in each suite (CaP-Agent0, OpenVLA, π0).
Robosuite bimanual handover: 20% (baselines) → 92% (Aspire). Easier contact-rich tasks remain near-saturated.
BEHAVIOR-1K navigate-and-pick-up-radio: 56% (CaP-Agent0) → 88% (Aspire); outperforms human-expert programs on both navigation and task success for both the soda-can and radio tasks.
Zero-shot transfer (library trained on LIBERO-90, evaluated on held-out LIBERO-Pro Long): 23% success on position perturbations, 38% on task perturbations; CaP-Agent0 saturates at 4% despite test-time reasoning + retries.
Library-size scaling: held-out LIBERO-Pro Long success increases monotonically as the skill library grows from empty to full.
Cross-embodiment real-robot transfer (sim-Franka → real bimanual YAM, Table 1):
- Put bowl on plate: 20/20 → 20/20, total tokens 8.65M → 5.11M.
- Lift soda can: 13/20 → 19/20, total tokens 61.94M → 6.58M (~9.4× reduction).
- Open/push drawer: 0/20 → 11/20 (no-skill baseline cannot find a successful program within a larger token budget).
Ablation [Fig. 6]: robot execution engine alone 14% → 62% macro-average on LIBERO-Pro; evolutionary search adds a further 10pp (62% → 72%).
Open-source V1.0 release numbers (reported in github.com/nvlabs/aspire README, 2026-10-16): 99% on 80 LIBERO-Pro Goal / Object / Spatial / Long tasks, 97–100% on every Robosuite task with Claude Opus 5.5, plus a real-world Rubik’s Cube solve on the YAM station driven by GPT-6 Astra and three additional distilled real-robot recipes (cup stacking, unscrew-and-pour, pill-in-bottle).

Why it’s interesting

ASPIRE is the full-paper version of the pattern that previously existed on the wiki only as a Jim Fan tweet (ASPIRE — self-evolving sensorimotor skill library that compounds across robots and the sim-to-real gap (Jim Fan / NVIDIA GEAR)), and it now sharpens three debates the wiki has been triangulating. First, against Playful Agentic Robot Learning (RATs; same Goldberg co-lead) which builds a skill library through a play stage driven by proposer/planner/diagnoser agents, ASPIRE builds it through evolutionary search over control programs conditioned on both successes and residual failure modes — the two papers now bound the “which inner-loop exploration mechanism scales best” question with concrete numbers (RATs: +20.6/+17.0pp on held-out tasks; ASPIRE: +77pp on LIBERO-Pro Object, 4% → 31% zero-shot LIBERO-Pro Long). Second, against the “learned experience library reduces required interaction depth” claim from Training-Free GRPO (Training-Free Group Relative Policy Optimization) in the text-tool-use setting, ASPIRE’s Table 1 cross-embodiment transfer gives the sharpest physical-world analog yet: a validated skill library cuts debugging tokens ~9.4× on soda-can lifting and enables a task (drawer manipulation) the no-skill baseline cannot solve within a larger budget — the same cost-asymmetry signature. Third, ASPIRE sits in the recursive-self-improvement cell alongside ENPIRE: Agentic Robot Policy Self-Improvement in the Real World ENPIRE (coding agent improves a VLA via physical RL), RoboRSI: Stable, Efficient, and Reusable Robot Self-Evolution in Complex Real-World Environments RoboRSI (four-role explicit attribution + live-environment validation), AIDE²: First Evidence of Recursive Self-Improvement AIDE² (outer loop optimizes the autoresearch harness itself), and Dream-RSI: Recursive Self-Improvement through Evolving Worlds Dream-RSI (gradient-free meta-opt via off-policy replay) — ASPIRE’s distinctive move is that the shipped artifact is a code library rather than a policy or model, and the sim-to-real / cross-embodiment transfer claim is validated at per-token cost not just per-task success. One caveat Luma researchers should carry: ASPIRE’s main results all depend on a frozen 1M-token-context frontier LLM (Claude Opus 4.6 at paper time, Claude Opus 5.5 / GPT-6 Astra at V1.0 release); the paper explicitly flags that smaller/weaker models have not been shown to sustain the loop, so the headline cost picture is tied to the frontier-API price curve.