RoboRSI: Stable, Efficient, and Reusable Robot Self-Evolution in Complex Real-World Environments

TL;DR

RoboRSI (Noematrix Lab + SJTU, Cewu Lu group) is a four-agent coding-harness for robot self-improvement whose structural bet is Top-Down Skill Refinement (TSR): decompose every task into a fixed four-level skill tree (Task Family → Compound Skill → Atomic Task → Base Skill), route execution outcomes back along the invoked path to localize the earliest failing node, and confine revisions to that node’s owning branch. On a real mobile manipulator the system develops multi-object household cleanup over 104 rounds; in simulation it reports the highest success rate on LIBERO, LIBERO-PRO, LIBERO-Plus, and RoboTwin, exceeding the strongest baseline by 2.7–11.0 percentage points. The explicit framing contrast is Code-as-Policies / Voyager / OpenETA: those recipes close a single execution-feedback loop, RoboRSI factors the loop into Manager / Planner / Engineer / Reviewer so cross-round failure attribution and validated skill release can run as distinct modules.

Key claims

The core architectural lever is a four-level skill tree with explicit scope + ownership, so a code-generating LLM’s local-detail bias stays aligned with the global task structure across repeated revisions [§ Method / TSR].
Four sub-agents own distinct lifecycle responsibilities: Manager decomposes tasks + maintains a queue over concurrent workers / resume points / skill versions, Planner forms plans against current observations + existing skills, Engineer calls tools and implements candidates, Reviewer diagnoses outcomes and routes revisions back to Manager [§ Method / four agents].
Attribution is routed along the actual call path to the earliest failing node and revision is confined to that node’s owning branch, rather than pushing workarounds downstream [§ Method / final feedback].
Once a branch stabilizes, the agent transitions from step-by-step tool selection to invoking consolidated Compound Skill code with parameterized objects / regions / spatial relations, and online agent reasoning resumes only when the environment changes or execution fails [§ Method / code consolidation].
Reported ablation on repeated tasks (120 tasks × 5 initial layouts, 600 episodes) with the same Base Skills: Code-on vs Code-off cuts median tokens by 29.4%, median VLM calls by 27.2%, and median wall time by 17.0% [§ Experiments / code consolidation panel].
A single corrective trajectory (304 frames, 1 h 4D hand-object + language retargeting) is converted via TSR into 8 training sequences / 2,432 samples / 1,000-step fine-tune for a downstream learning-based policy on libero_spatial_swap/0 [§ Experiments / learning-based policy].
Under a stratified perturbation panel of 840 instances (7 categories × 120) on LIBERO base, RoboRSI reports +137 passed instances after further revision rounds across visual-texture / camera-viewpoint / language-instruction / lighting / object-layout / robot-initial-state / sensor-noise axes [§ Experiments / perturbation panel].
Headline reported sim results: SOTA on LIBERO, LIBERO-PRO, LIBERO-Plus, and RoboTwin, exceeding the strongest baseline by 2.7–11.0 percentage points (per paper abstract); real mobile-manipulator household cleanup developed over 104 rounds [§ Abstract].
Design-principle claim with iteration-time metric: a revision counts as successful not only when the current task succeeds but when repeated execution, extra verification, model fallback, human intervention, and code branching all decrease across subsequent iterations [§ Method / correct attribution].

Method

RoboRSI builds a continuous closed loop around TSR, a four-level skill decomposition (Task Family → Compound Skill → Atomic Task → Base Skill) that fixes the task structure at the top and places the concrete capabilities below in scoped nodes with explicit input/output contracts. Four sub-agents share this tree: on a new request the Manager decomposes it into Atomic Tasks and identifies required Base Skills; the Planner produces an executable plan per Atomic Task; the Engineer implements code-based policies and calls tools against the live robot (or sim); the Reviewer turns the physical outcome into a task summary and a revision proposal, routed back to the responsible Manager / Engineer branch for diagnosis + focused tests + the next revision. Human-Friendly Steering concentrates human input on objectives + direction + safety; Agent-Friendly Structure constrains the coding model to local, ownership-scoped edits.

The paper names four load-bearing design principles: (a) clear responsibility + mutual trust across skills so downstream nodes verify applicability of mature upstream results rather than re-implement them; (b) correct attribution + convergent iteration — repair the module that owns a failure instead of pushing a workaround downstream, and judge revisions on whether repeated execution / verification / fallback / human intervention / code branching decrease over iterations; (c) program parameterization + windowed repair — pixels / coordinates / poses are run-local and don’t enter the reusable program, and routine code revisions are evaluated against a history replay window of recent successes + failures to avoid breaking previously-working behavior; (d) human-knowledge distillation — each human correction is consolidated into automatic checks / rules / tests / documentation rather than left in conversation history. When a branch stabilizes, a Compound Skill is extracted with objects / regions / spatial relations as parameters; the agent shifts from step-by-step tool selection to selecting + monitoring the Compound Skill while consolidated code handles repeated tool steps, with online agent reasoning only resuming on environmental change or failure.

Results

LIBERO / LIBERO-PRO / LIBERO-Plus / RoboTwin: paper reports SOTA across all four with +2.7 to +11.0 pp over the strongest baseline [§ Abstract].
Real mobile-manipulator household cleanup: 104 rounds of self-evolution from a reported engineering baseline of ~2,000–3,000 task-specific lines + two engineers × one month in 2024 down to one capability-iteration day with RoboRSI (reset / outcome check / log reading / code diagnosis absorbed into the four agents) [§ project page framing].
Code consolidation ablation (120 tasks × 5 initial layouts, 600 episodes, same Base Skills): Code-on vs Code-off reduces median tokens −29.4%, median VLM calls −27.2%, median wall time −17.0% [§ Experiments].
Perturbation recovery (840-instance stratified panel, 7 × 120): +137 passed instances after further revision rounds on top of the LIBERO base [§ Experiments].
Learning-based policy from TSR traces: one corrective trajectory (304 frames) → 8 training sequences → 2,432 samples → 1,000-step fine-tune on libero_spatial_swap/0 [§ Experiments].

Abstract-level numbers only for the headline SOTA — per-benchmark splits + baseline identities + CI widths not retrievable from the arxiv abstract at filing time; project page carries interactive LIBERO / RoboTwin viewers and the perturbation panel.

Why it’s interesting

RoboRSI sits in the explicit intersection of three threads this wiki tracks. On the orchestration-over-frozen-VLA axis of VLA Models, it is the sharpest filed instance to move the lever from a single orchestrator wrapping a single learned policy (as in Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents Harness VLA or Addressing the Orchestration Gap in Generalist Robots via Physical Agency (Pigey) Pigey) to a four-agent decomposition where the policy is synthesized code emitted by the Engineer against a tree-structured skill vocabulary — closer in spirit to Playful Agentic Robot Learning RATs’s Code-as-Policy skill library but with the Reviewer formalizing failure attribution and the TSR scope confining where revisions may land. On Agentic Software Engineering it imports the “harness is the lever” thesis directly into robot control: the four roles mirror SWE agent-manager / planner / coder / critic patterns but with the robot’s physical trace as the authoritative evaluator, giving a concrete datapoint for whether coding-agent scaffolds generalize off text-only CLI benchmarks. And on AI-for-AI Research it adds a fifth cell distinct from AIDE² / Dream-RSI / SoL-H3 / SoL-Pi: the “autoresearch” loop is over physical-skill code and the validation gate is live-environment execution rather than a sim scoreboard or a model-weight-benchmark pair — a tighter counter-recipe to pure sim-eval RSI with per-rollout environmental non-stationarity as both cost and realism.