Recova: Agent-Guided Failure Recovery for Autonomous Robotic Manipulation

TL;DR

Recova (NVIDIA + UCSD + UT Austin) attacks manipulation-failure recovery by co-developing a task policy and a recovery policy in a reconstructed digital twin under a single agent, then verifying and refining both through real-world experience. In the twin, the agent diagnoses failures, writes and tests corrective programs, and collects successful task- and recovery-rollout data for two separate policies. At deployment the same agent monitors progress, invokes a learned or programmatic recovery when execution stalls, verifies scene restoration, and resumes task execution; when no suitable recovery exists, a human demonstration resolves the failure and feeds back into training. Physical rollouts and human demonstrations are routed to the matching policy for DAgger training, turning failures into reusable recovery capabilities and progressively driving human-intervention rate toward zero.

Key claims

Jointly developing a task policy and a recovery policy in a reconstructed digital twin, with the agent diagnosing failures and testing corrective programs, outperforms strong baselines across six LIBERO-Pro settings (78.8% mean success vs 71.7%) and four MolmoSpaces categories (64.9% vs 38.0%) [§Abstract].
Parallel data collection across four real-robot workstations with DAgger fine-tuning raises mean success from 23.8% to 77.5%, with recovery skills raising it further to 87.5% [§Abstract].
Agent-triggered recovery during deployment follows a monitor-invoke-verify-resume loop: the agent decides when execution has entered a failure state, invokes either a learned or programmatic recovery, verifies that the scene has been restored to a resumable state, and only then returns control to the task policy [§Abstract].
Recovery capabilities compound over collection rounds: over four rounds on one task, observed human-intervention rate falls from 87.5% to 0% as new recoveries get added to the system’s library and the next deployment needs them less often [§Abstract].
When the system has no suitable recovery available, a human demonstration resolves the failure and enters the learning loop, which is how the recovery library expands over time [§Abstract].

Method

The pipeline runs two coupled loops. The twin loop reconstructs the deployment scene as a digital twin, where a central agent executes the current task policy, diagnoses failure modes when execution stalls (classifying the stall and identifying what corrective behavior would restore the scene), and writes/tests corrective programs. Successful task and recovery rollouts in the twin are collected into two separate datasets, used to train the task policy and the recovery policy respectively. The real loop deploys both policies on physical hardware with the agent in a monitoring role: during execution the agent polls progress, and on failure invokes either a learned recovery policy or a programmatic corrective (the twin-authored programs double as deployment-time primitives). After the recovery runs, the agent verifies scene restoration before returning control to the task policy. When neither learned nor programmatic recoveries resolve the failure, a human demonstration completes the recovery; that demonstration is then routed back to the recovery dataset for DAgger training, closing the loop. Parallel collection is run across four real-robot workstations so DAgger data accumulates faster than single-arm pipelines allow.

Results

LIBERO-Pro / MolmoSpaces (sim). Six LIBERO-Pro settings: 78.8% Recova mean vs 71.7% strongest baseline. Four MolmoSpaces categories: 64.9% vs 38.0% — a wider margin than LIBERO-Pro, consistent with MolmoSpaces being less saturated by VLA baselines at filing [§Abstract].

Real-robot DAgger (four workstations in parallel). Mean success 23.8% → 77.5% from DAgger fine-tuning alone; adding recovery-skill invocation at deployment time lifts to 87.5% [§Abstract].

Per-task compound effect over collection rounds. Over four rounds on a single task, human-intervention rate falls 87.5% → 0% — the sharpest filed number for an agent-guided-recovery pipeline on the “capabilities compound over rounds” axis [§Abstract].

Why it’s interesting

Recova stakes out a new cell on the recipe-lever board at VLA Models that is distinct from every existing failure-recovery entry: it co-develops the task and recovery policies in a reconstructed digital twin with the agent authoring both corrective programs and learned-policy training data, rather than treating recovery as a separable capability trained from a different data source. The closest filed neighbors are EgoRecovery: Acquiring Failure Recovery Ability Through Human Recovery Demonstration (which collects recovery data from off-robot egocentric human capture and uses a gated corrective-intent module), Enact launch — post-training infrastructure that generates targeted recovery data for robotics VLAs Enact and Empirical results fine-tuning π0.5 on a real manufacturing task (which push human-in-the-loop DAgger on real robots at single-task scale), and FailSafe: Reasoning and Recovery from Failures in Vision-Language-Action Models FailSafe (a companion VLM emits recovery deltas against a frozen VLA at runtime). Recova is the first to put all four pieces in one loop: twin-based diagnosis-plus-corrective-program authoring, twin-based data generation for both the task and recovery policies, agentic deployment-time monitor-invoke-verify-resume, and real-world DAgger that routes physical rollouts and human demos to the matching policy.

It also connects naturally to Simulator-in-the-Loop Refinement — the digital twin here is a training-time simulator used to generate recovery capabilities offline rather than a per-step refinement loop like SILR or SIMPACT, but shares the “agent authors corrective programs in a physics simulator” shape with SIMPACT: Simulation-Enabled Action Planning using Vision-Language Models SIMPACT, with the critical difference that SIMPACT keeps the loop at inference time per episode while Recova amortizes the authoring work across many deployments. The intervention-scaling thesis of MicroFactory — 99.9% reliability via $5 human-in-the-loop DAgger retraining on Jetson (Ilir Aliu × Igor Kulakov podcast) MicroFactory and Robo Robotics launch — Robo-T bimanual humanoid, sub-$10/hour with Roboport teleoperation platform (Kyle Noble) Robo Robotics — that per-intervention cost and intervention rate dominate deployment economics — gets a concrete 87.5% → 0% datapoint over four collection rounds.