RE-0: Verified Recursive Improvement of Embodied Code-as-Policy Agents through Local On-Policy Distillation

TL;DR

RE-0 recasts self-improvement of embodied Code-as-Policy agents (LLMs that execute long-horizon manipulation by writing and running robot programs) as a locate–verify–weight recursion over student-induced failure histories. A diagnostic agent localizes the earliest plausibly-causal failure boundary in a student rollout; a teacher emits a minimal repair contract; the environment is restored to a saved snapshot and the correction is executed against the student baseline from the same state under shared randomness; only interventions whose paired counterfactual advantage passes a lower-confidence-bound gate become supervision for on-policy distillation (RE-OPD). A finite-sample bound shows per-round student gain is lower-bounded by verified intervention gain up to verification + projection error. Across six long-horizon robosuite tasks, 3–67 verified rows lift base-model success from 4–68% to 62–100%, with the LCB admission gate — not advantage weighting — identified as the load-bearing variable; a 76%→58% regression on one task is caught by the stage-entry criterion rather than silently propagated.

Key claims

RE-0 defines local teacher credit on student-induced histories as the expected counterfactual advantage of a bounded teacher correction over the current student from the same snapshot, under shared randomness, with control returning to the student after the segment — not teacher trajectory return or global success rate [§3.1, Eq. 1].
Repair contracts are proposed by a teacher diagnostic in monotonically-increasing scope and verified in order; only the smallest verified scope is admitted, so “minimal necessary repair” is the smallest scope passing the gate rather than the shortest candidate string [§3.2].
The counterfactual verifier is a paired same-state LCB with two-layer confidence radius controlling both teacher-candidate and rollout sampling error, allocated across the whole recursion via a Boole union bound for simultaneous coverage [§3.2, Eq. 2].
A performance-difference-lemma argument shows the hybrid policy’s expected gain satisfies $J(\pi^{\mathrm{hyb}}) \ge J(\pi_s) + G_{\mathrm{int}} - \epsilon_{\mathrm{verif}}$ with high probability, where $G_{\mathrm{int}}$ is the verified intervention gain and $\epsilon_{\mathrm{verif}}$ collects snapshot-restoration and execution-identity errors [§3.2, Eq. 4].
RE-OPD distills the verified local hybrid into the student via a KL projection that factors exactly into a student-retention term plus a credit-weighted teacher-projection term, where supervision weight $\lambda(h) = \alpha \cdot \pi^{\mathrm{int}}(h) \cdot \hat A_{\mathrm{LCB}}(h)^+$ is the verified counterfactual advantage [§3.3, Eq. 5].
The main theorem (Theorem 1): under bounded returns + valid contracts + simultaneous-coverage LCB + bounded occupancy-weighted projection error, per-round student gain is $\Delta J \ge G_{\mathrm{int}} - \epsilon_{\mathrm{verif}} - \epsilon_{\mathrm{proj}}$ with probability $\ge 1 - \delta$ , no global teacher dominance required [§3.3, Eq. 6].
Cold-start (RE-OPD-cold, teacher in the seat) and refinement (RE-OPD-refine, student in the seat) share all machinery; the guarantee transfers verbatim regardless of who occupies the seat, so even a teacher that cannot complete the task contributes trajectories inheriting verified self-improvement credit [§3.4, Proposition 1 and Corollary 1].
The load-bearing variable is verified corpus composition, not weighting: on the lift journal across three trainer seeds, LCB admission beats unfiltered (6/50, 2/50, 3/50), sign-only ( $A > 0$ ) selection (5/50, 4/50, 4/50), and outcome-based teacher-program selection (5/50, 9/50, 4/50) with 11/50, 5/50, 13/50 — all with the identical training recipe [§4.4, Table 2].
Cold-start lifts base-model success from 4–68% to 62–100% with just 3–67 verified rows across six tasks (restack 62% / 3 rows, lift 76% / 7 rows, nut 88% / 27 rows, spill 100% / 67 rows, door 100% / 40 rows, cubestack 100% at saturation / no training) [§4.2, Table 1].
Row count is not monotonic with gain — restack reaches 62% on 3 rows while nut needs 27 — the decisive factor is per-row verified credit, not corpus size [§4.2].
The same gate used as a measurement instrument produces a monotone teacher-effect ladder before any training: student self-play 0/34 admitted, narrow local experts 0/32 admitted (10% rescue ceiling + deterministic contract-format failures), strong API teacher 4 admitted on 13 events (19–47% rescue rates) [§4.3, Table 6].
The strict LCB gate costs no sensitivity: admitted events cluster at $\hat A \ge 0.25$ while abstained events sit at $\hat A \le 0.08$ , and sweeping the threshold 0.24 → 0.01 leaves the admitted set unchanged [§4.3, Fig. 3c-d].
The gate is two-sided: one API-emitted patch dropped the baseline 8/16 → 1/16 ( $\hat A < -0.3$ ) and was rejected — credit assignment is a sign decision on the intervention’s causal effect, not one-directional imitation [§4.3].
GRPO with terminal rewards fails on the same tasks: entropy collapses from 0.30 → 0.002 and transfers 0–1/50 on non-saturated tasks vs base 8%/68%/4%, retaining only the already-saturated task — sparse terminal reward cannot localize the harmful decision inside a long program [§4.3].
The outer acceptance criterion catches a regression: lift admits one event with thin verified credit ( $A \approx 0.03$ ), the projection regresses from 76% to 58% with branch collapse (reproduces at seed 2, 76% → 46%), and the stage-entry check halts the recursion rather than propagating the regressed checkpoint — a measured safety property, not just an upper bound [§4.2, Remark 1].
Weight-scale and normalization ablations on a fixed verified corpus cannot recover the gain and can induce catastrophic forgetting — isolating the load-bearing variable to what enters the corpus, not how the entered rows are weighted [§4.3, Appendix E].
The distilled teacher-free student transfers to a dual-arm AgileX PiPER platform zero-shot with no hardware-specific policy edits: 13/20 restack, 12/20 nut, 20/20 spill, 20/20 cubestack, 2/20 lift; hardware deployment makes no teacher calls [§4.5, Fig. 4].

Method

RE-0 instantiates the locate–verify–weight recursion in three coupled modules operating on student-induced trajectories.

Localization. For each failed rollout $\tau \sim \pi_s^{(g)}$ , a student-side diagnostic agent inspects runtime exceptions + unrealized expected effects + other structured evidence and outputs the earliest candidate failure history $h^\star$ with its full simulation snapshot $\sigma(h^\star)$ . If no causally-consistent executable boundary is found, no teacher call fires. A teacher-side diagnostic then emits repair contracts $\{c_1, c_2, \ldots\}$ in increasing scope cost — each contract fixes the repair boundary, the allowed program symbols and API surface, and the maximum segment length.

Verification. For each contract $c_i$ in increasing scope order, the environment is restored to $\sigma(h^\star)$ . Under shared downstream randomness $\xi$ , the contract-constrained teacher distribution $\pi^{\mathrm{int},c_i}$ and a student resample $\pi_s$ are executed, control returns to the student after the segment completes, and the paired counterfactual advantage $\hat A(h^\star, c_i)$ is the $\xi$ -average of return differences minus a scope cost $\lambda_c \cdot \mathrm{scope}(c_i)$ . A two-layer confidence radius $\mathrm{LCB}_{c_i}(h^\star) = \hat A - r_{\mathrm{cand}} - r_{\mathrm{roll}}$ bounds both teacher-candidate and rollout sampling error; a Boole union bound allocates radii across all verified histories in the whole recursion for simultaneous coverage at level $1 - \delta$ . The smallest contract passing both LCB $> A_{\min}$ and the execution-safety / program-context / snapshot-identity predicate is admitted. Rewind semantics: the live attempt that localized $h^\star$ and the verification pass from the snapshot share the same snapshot, so the first pass is a common data-collection cost that cancels in the per-round bound.

Projection. The verified local hybrid $\pi^{\mathrm{hyb}} = (1 - p^{\mathrm{int}}) \pi_s + p^{\mathrm{int}} \pi^{\mathrm{int}}$ is projected into a teacher-free student via a KL objective that factors into a student-retention term + a credit-weighted teacher-projection term (Eq. 5), with weight $\lambda(h)$ equal to the verified LCB advantage. The teacher is black-box (no conditional distributions), so RE-OPD uses the trajectory-level hard estimator — the loss is standard SFT weighted by verified credit, with the identical optimizer family as a plain SFT baseline. The difference from vanilla SFT is not the loss form but what enters it: supervision locations, per-event weights, and the retention term are decided by environment verification, not teacher sampling. After each projection, trajectories are regenerated with the new student and credit is re-allocated on the new occupancy. Termination is structural — if the gate closes almost everywhere (headroom exhausted or teacher effect below the verification noise floor $\epsilon_{\mathrm{verif}}$ ), Eq. 5 degenerates to retention and the recursion safely abstains.

Measurement guard (Remark 1). The per-round bound guards the credit level; a separate deployment criterion compares a held-out stage-entry evaluation to a threshold $\tau$ and halts if the projection regresses. This is what catches the lift 76% → 58% branch collapse at seed 1 and 76% → 46% at seed 2.

Results

Six long-horizon robosuite tasks (restack, spill, nut, lift, cubestack, door) spanning single/dual arms, rigid/articulated objects, coarse/fine manipulation. Base-model success spans 4% (lift, nut) to 100% (cubestack). Held-out eval = 50 paired seeds per task with Wilson 95% CIs; collection seeds always disjoint [§4.1].
Flash-API teacher as single-shot primary policy (no RE-0 loop): 4%/68%/4%/12%/50%/50% across tasks — flat on restack/lift where RE-0 lifts to 62%/76% [§4.2, Table 1].
RE-OPD-cold: restack 4% → 62% with 3 rows (3/50 → 31/50); spill 68% → 100% with 67 rows (34/50 → 50/50); nut 4% → 88% with 27 rows (2/50 → 44/50); lift 4% → 76% with 7 rows (2/50 → 38/50); door 96% → 100% with 40 rows (48/50 → 50/50); cubestack already saturated at 100%, exempted (no-train) [§4.2, Table 1].
RE-OPD-refine on restack lifts 62% → 80%; abstention on spill (already at 100%, admissions empty, Corollary 2 fires at runtime) [§4.2].
Admissions decay with occupancy — 6/6 at round $g=1$ vs 4/15 at $g=3$ on the restack recursion [§4.2, Fig. 3b].
Negative result captured by the guard: lift refinement admits one event with $A \approx 0.03$ , projection drops 76% → 58% at seed 1 (branch collapse on resample variability) and 76% → 46% at seed 2; the stage-entry measurement halts recursion and discloses the regressed checkpoint [§4.2, Remark 1, Appendix D note (d)].
Admission ladder as teacher-effect measurement before any training: student self-play (34 cycles) → 0 admissions; narrow local experts (32 events) → 0 admissions, 10% rescue ceiling + deterministic contract-format failures; strong API teacher → 4 admissions on 13 events, per-event rescue rates 19–47% [§4.3, Table 6].
Threshold insensitivity: admitted events cluster at $\hat A \ge 0.25$ , abstained at $\hat A \le 0.08$ ; the admitted set is unchanged across threshold sweep 0.24 → 0.01 [§4.3, Fig. 3c-d].
Gate rejects harm: one API patch dropped baseline 8/16 → 1/16 ( $\hat A < -0.3$ ) and was correctly rejected [§4.3].
GRPO control on same environments: entropy 0.30 → 0.002, transfers 0–1/50 on non-saturated tasks against base 8%/68%/4%, retains only the saturated task [§4.3].
Selection-rule ablation on lift across three trainer seeds (same journal, same recipe): LCB (3 rows) 11/50, 5/50, 13/50, mean $A$ 9.7; sign-only (7 rows) 5/50, 4/50, 4/50, mean 4.3; unfiltered (25 rows) 6/50, 2/50, 3/50, mean 3.7; outcome-based teacher-program selection (27 rows) 5/50, 9/50, 4/50, mean 6.0 [§4.4, Table 2].
Objective-composition quartet on restack (separate 50-seed cohort): stripping the student prefix (patch-only) or the retention term (weighted) collapses the gain the full objective and uniform credit largely retain — the full exact-mixture projection and uniform-on-the-verified-corpus SFT fall within trainer-seed variance of each other, isolating verified-admission (the corpus) as load-bearing and weighting as secondary [§4.4].
Transfer: cross-task eval is selective rather than universal (restack training improves nut, spill’s wiping dynamics do not carry); pooled multi-task training shows substantial interference [§4.5, Appendix F].
Zero-shot real-robot deployment on dual-arm AgileX PiPER (SAM3 open-vocabulary perception, no hardware-specific edits, no teacher calls): 13/20 restack, 12/20 nut, 20/20 spill, 20/20 cubestack, 2/20 lift [§4.5, Fig. 4].

Why it’s interesting

RE-0 is the sharpest formal statement to date of the “verify teacher corrections per-event before letting them in” position that the embodied-coding-agent cluster has been circling. Two framings make it a natural reference point.

First, it is a structural counter-recipe to the two dominant post-training threads on robot-code policies. Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe identifies two necessary conditions for on-policy distillation to help a text LLM (thinking-pattern compatibility + genuinely new teacher knowledge) and shows OPD breaks silently when they fail. RE-0 extends that failure-mode taxonomy into embodied, history-dependent, long-horizon execution where a single harmful code segment alters the environment state, the program namespace, and the subsequent observable history — and gives the first principled answer: don’t check teacher strength at the global level, check per-event local counterfactual advantage on student-induced histories, and absorb only what passes a paired same-state LCB gate. The empirical sharpening that the LCB admission gate carries the gain while advantage weighting is replaceable by uniform weights (within trainer-seed variance) is a direct contribution the OPD literature has not had: it says the load-bearing variable is corpus composition under verified counterfactual admission, not reward shape.

Second, it stakes out an orchestration-cell in the recipe-lever board tracked under VLA Models that none of the existing frozen-base-plus-companion recipes occupy — distinct from FailSafe: Reasoning and Recovery from Failures in Vision-Language-Action Models FailSafe (10-step polling companion VLM emitting runtime deltas), Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents Harness VLA (LLM planner invoking narrow retryable primitives at subtask boundaries), and the Enact launch — post-training infrastructure that generates targeted recovery data for robotics VLAs Enact / MicroFactory — 99.9% reliability via $5 human-in-the-loop DAgger retraining on Jetson (Ilir Aliu × Igor Kulakov podcast) MicroFactory on-robot DAgger lineage. RE-0’s load-bearing move is measurement discipline on the correction signal before training: the environment-grounded LCB on paired same-state rollouts is a decision procedure for whether a specific teacher intervention produced a positive causal contribution, and only verified events — a tiny verified corpus (3–67 rows across six tasks) — survive into SFT. The reported real-robot zero-shot transfer at verified-event counts matching single-digit sim journal entries is the sharpest filed datapoint on the “verify then compress” axis in this cluster, and the GRPO control (0–1/50 under sparse terminal reward on the same tasks) is a crisp quantitative statement of why outcome-RL cannot localize credit inside a long code trajectory.

The guardrail finding is independently useful: the outer acceptance criterion catches a 76% → 58% regression on lift and halts the recursion, with the regression reproducing at a second trainer seed. This is the first filed empirical demonstration in the embodied-agent cluster that per-round guarantees + stage-entry measurement are individually necessary and jointly sufficient to make self-improvement safe — a template Reasoning RL‘s on-policy distillation lineage (On-Policy Distillation, Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy Distillation, MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training) could plausibly import.