Skip to content
Bud's Wiki

EgoRecovery: Acquiring Failure Recovery Ability Through Human Recovery Demonstration

TL;DR

Section titled “TL;DR”

EgoRecovery treats failure recovery as its own scaling problem: collecting recovery trajectories via robot teleoperation is dominated by the cost of setting up diverse failure states and resetting the environment, while a human operator wearing egocentric capture can arrange failure configurations at task level and record short corrective segments >10× faster per hour. To bridge the human–robot embodiment gap, the paper introduces a compact corrective-intent latent space shared by human and robot recovery data — capturing timing and magnitude of correction rather than raw motion — and shows that a small amount of robot recovery data is enough to connect this intent to executable actions. At deployment a learned recovery gate predicts, from robot observations, when correction is needed and activates the corrective intent only in recovery states. On real-world tasks it improves success-from-failure-start over robot-only recovery, direct co-training with raw human recovery data, and direct intent-transfer baselines.

Key claims

Section titled “Key claims”
  • Recovery data is intrinsically more expensive to teleoperate than success data because failure induction, corrective execution, and reset all consume operator time; the diversity of failure modes further widens the required data budget vs. success demonstrations [§Abstract].
  • Egocentric human capture with task-level failure arrangement and short recovery segments yields >10× as much valid recovery data per hour as robot teleoperation under the paper’s protocol [§Abstract].
  • A compact corrective-intent space — a shared latent that encodes when and how strongly to correct, rather than raw joint or wrist trajectories — is the interface used to align human and robot recovery data; only a small number of robot recovery demonstrations are needed to connect this intent to executable robot actions [§Abstract].
  • A learned recovery gate predicts from robot observations alone when a correction is needed and activates the corrective intent only in recovery states, keeping normal task execution untouched [§Abstract].
  • EgoRecovery beats three baselines on real-world recovery-from-failure-start success: robot-only recovery training, direct human+robot co-training on raw recovery data, and direct intent-transfer without gated activation [§Abstract, Results].

Method

Section titled “Method”

The pipeline has three stages. First, egocentric human demonstrators arrange task-level failure configurations and record short recovery segments; the protocol is designed so that most operator time is spent recording valid recovery motion rather than resetting scenes or inducing failures, giving the reported >10× rate over robot teleoperation. Second, the human recovery clips and a small robot recovery dataset are co-trained via alignment into a shared corrective-intent latent that captures the timing and magnitude of correction, deliberately abstracting away embodiment-specific motion so that the human/robot mismatch is not directly modeled at the joint or wrist level. Third, a recovery gate is trained on robot observations to predict when the current rollout has entered a recovery state; the corrective-intent module is then activated only when the gate fires, so success-mode execution is not perturbed by the recovery module.

Results

Section titled “Results”

Real-world evaluation on recovery-from-failure-start tasks reports absolute wins over three baselines: (a) a robot-only recovery-data policy, (b) direct co-training on the union of robot and raw human recovery demonstrations, and (c) direct intent-transfer without the gated activation mechanism. The paper positions the >10×-per-hour human-vs-teleop recovery-data rate as the load-bearing empirical claim on the data side, and the recovery-gate + corrective-intent design as the load-bearing claim on the modeling side.

Why it’s interesting

Section titled “Why it’s interesting”

EgoRecovery stakes out a new interface position on the human-to-robot retargeting design axis mapped by Human-to-Robot Retargeting: don’t retarget human motion (as in Translation as a Bridging Action: Transferring Manipulation Skills from Humans to Robots or Do as I Do: Dexterous Manipulation Data from Everyday Human Videos), and don’t retarget generic task intent (as in LUCID: Learning Embodiment-Agnostic Intent Models from Unstructured Human Videos for Scalable Dexterous Robot Skill Acquisition) — instead retarget a corrective intent narrowly scoped to failure-recovery states, gated on by a learned trigger. The scope narrowing is what makes the small-robot-data recipe plausible: the shared latent only has to represent the “when + how strongly” of correction, not the full motion manifold. This is also the first filed paper to attack failure recovery as a data-generation problem rather than a policy-conditioning problem — complementing UniIntervene: Agentic Intervention for Efficient Real-World Reinforcement Learning‘s HiL-RL reframing (agentic intervention selects when a human should correct online) with a purely offline, capture-then-align recipe. The >10× rate claim is a direct data-side counterpart to HumanScale: Egocentric Human Video Can Outperform Real-Robot Data for Embodied Pretraining‘s matched-volume result that egocentric human video can outperform real-robot data — extended here from success to recovery.

See also

Section titled “See also”