Skip to content
Bud's Wiki

FailSafe: Reasoning and Recovery from Failures in Vision-Language-Action Models

TL;DR

Section titled “TL;DR”

FailSafe is a sim-native pipeline that automatically generates paired failure trajectories and executable recovery actions by perturbing motion-planner sub-goals in ManiSkill with three fundamental failure modes (translation, rotation, no-ops), then verifying via replay that each candidate corrective 7-DoF delta actually resolves the failure and lets the task finish. Fine-tuning LLaVA-OneVision-7B on the resulting 131k failure–action + 56k success dataset produces FailSafe-VLM, a companion model that a base VLA queries every 10 steps: if a failure is detected, FailSafe-VLM emits the failure type plus an executable recovery delta and the VLA resumes. Across three ManiSkill tasks it improves π0-FAST +4 pp, OpenVLA-OFT +8 pp, and OpenVLA +22.6 pp on average, and the same checkpoint transfers to unseen objects (+17.4 pp) and to an unseen xArm 6 embodiment (+6.7 pp) without any retraining. Notably, FailSafe-VLM’s training camera view is deliberately misaligned with the base VLA’s — at evaluation FailSafe-VLM shares the VLA’s view (novel to it), making the reported gains a viewpoint-generalization result as well.

Key claims

Section titled “Key claims”
  • The failure taxonomy — translation (x/y/z perturbations), rotation (roll/pitch/yaw), and no-ops (frozen arm) — is claimed to cover “the majority of failure scenarios observed in VLA control,” injected at motion-planner sub-goal boundaries with random magnitudes so a single stage’s pose becomes p̃_k and the full rollout p_1, …, p̃_k, …, p_N closely mimics VLA-deployment failure modes [§III-A].
  • The load-bearing distinction from AHA / RoboFAC / REFLECT is that FailSafe outputs executable 7-DoF recovery deltas rather than textual explanations, computed as the 7-DoF pose difference between a deviated-trajectory pose p̃_k and a candidate corrective pose from the ground-truth trajectory, constrained to windows that avoid gripper–object collisions [§III-B].
  • Systematic verification is the mechanism that guarantees dataset quality: each (p̃_k, p_corr) pair is replayed in-sim (arm → p̃_k → p_corr → remaining GT poses), and only pairs whose replay successfully completes the task are added to the dataset [§III-C].
  • FailSafe-VLM beats frontier VLMs at failure reasoning by a wide margin: on 1,712 held-out test entries, 0.9094 binary-success / 0.8368 failure-type accuracy / 0.6522 cosine similarity vs. GPT-4o 0.7007 / 0.1960 / 0.0117 and Gemini-2.5-flash 0.6229 / 0.1412 / -0.0121 — >4× accuracy at identifying the specific failure mode [Table II].
  • Attaching FailSafe-VLM as a 10-step-interval assistant improves all three tested VLA baselines on three ManiSkill tasks: OpenVLA 14.7% → 37.3% (+22.6 pp), OpenVLA-OFT 90.7% → 98.7% (+8.0 pp), π0-FAST 78.7% → 82.7% (+4.0 pp) — weakest baselines benefit most [Table III].
  • Cross-object generalization: FailSafe-VLM trained on Cube tasks assists OpenVLA-OFT on Sphere / Charger tasks, 53.3% → 70.7% average (+17.4 pp), attributed to shared failure-mode structure across manipulation tasks rather than object-specific reasoning [Table IV].
  • Cross-embodiment generalization: the same FailSafe-VLM checkpoint (trained on Franka Emika Panda) transfers to an xArm 6 without retraining, lifting stack-cube 56% → 76% (+20 pp) with no degradation on the other two tasks (+6.7 pp average) [Table V].
  • FailSafe-VLM is trained under a distinct “front / side / hand” three-view configuration, but evaluated on a fourth view aligned with the base VLA’s training angle — so the gains on the joint system are also evidence that the failure-reasoning capability generalizes across camera viewpoints [§III-D, IV-B].
  • The fine-tuning recipe co-trains on FailSafe + a RoboPoint VQA mixture “to improve generalization,” using LLaVA-OneVision-7B initialized from a single-image checkpoint with SigLIP vision tower, joint fine-tuning of vision + adapter + LM at 1e-5 (2e-6 for vision) over one epoch on 32 H100s [§III-E].

Method

Section titled “Method”

FailSafe wraps around a motion-planned simulator (ManiSkill in this instance, but the pipeline is designed to transfer to any simulator that supports motion planning). Failure generation writes a YAML config listing per-stage failure modes, noise ranges, and injection points, and a custom environment wrapper randomly perturbs the pre-defined pose at one stage per trajectory to produce a deviated rollout p_1, p_2, …, p̃_k, …, p_N. If the task fails as a result, the images, failure trajectory, and failure type get passed on. Action collection then searches, for each p̃_k after step 10 of the failure trajectory, a corresponding corrective pose p_corr drawn from a mid-window of the ground-truth trajectory (10 steps in from start, 3 steps before end) — the constraint that prevents recovery actions from causing gripper–object collisions. Multiple candidate (p̃_k, p_corr) pairs are sampled per trajectory, and the corrective action is the 7-DoF pose delta. Systematic verification replays each pair through the motion planner, keeping only those whose replay ends in task success.

The dataset itself is 131k failure–action pairs across Pick Cube / Push Cube / Stack Cube spanning translation, rotation, and no-ops modes, plus ~56k ground-truth (no-failure) trajectories so the model can discriminate success from failure. Each entry contains 10 consecutive image observations of the robot trajectory from three camera perspectives (front / side / hand). The Q/A format asks the model — given image observations + task instruction — to output current sub-task, whether a failure occurred, and (if yes) the failure type + corresponding recovery action. FailSafe-VLM is obtained by full instruction-fine-tuning LLaVA-OneVision-7B (SigLIP + Qwen2-7B-Instruct + 2-layer GELU MLP adapter) co-trained with the RoboPoint VQA mixture, one epoch on 32 H100s.

At deployment FailSafe-VLM is architected as an external assistant running alongside a base VLA on the same physical camera stream. Every 10 steps the base VLA yields control; FailSafe-VLM inspects the last 10 frames and either passes (no failure detected) or outputs a recovery delta that the robot executes directly. Control then returns to the base VLA. The training-time three-view configuration is deliberately different from evaluation-time single-view, so FailSafe-VLM is evaluated on a camera view it was not trained on — matching the realistic deployment constraint where an extra dedicated camera is unavailable.

Results

Section titled “Results”

Failure-reasoning capability on held-out spatial configurations: FailSafe-VLM 0.9094 / 0.8368 / 0.6522 across binary-success / accuracy / cosine-similarity vs. GPT-4o 0.7007 / 0.1960 / 0.0117 and Gemini-2.5-flash 0.6229 / 0.1412 / -0.0121; Qwen2.5-VL degenerates to always predicting “no failure,” yielding 0.2401 binary-success and 0 cosine [Table II].

VLA-assist gains on three ManiSkill tasks (Pick Cube / Push Cube / Stack Cube, per-task success then average): π0-FAST 88 / 52 / 96 → 88 / 64 / 96 (78.7 → 82.7, +4.0 pp); OpenVLA 28 / 4 / 12 → 48 / 24 / 40 (14.7 → 37.3, +22.6 pp); OpenVLA-OFT 84 / 88 / 100 → 96 / 100 / 100 (90.7 → 98.7, +8.0 pp) [Table III]. The magnitude of the OpenVLA improvement is disproportionate — that VLA’s baselines are the weakest on the panel, and FailSafe-VLM’s corrective deltas are large enough to move it from near-failure to competitive across the board.

Object-generalization on Pick Sphere / Place Sphere / Pick Charger with OpenVLA-OFT: 44 / 36 / 80 → 68 / 52 / 92, 53.3 → 70.7 average (+17.4 pp) [Table IV]. Embodiment-generalization on xArm 6 with OpenVLA-OFT: 100 / 100 / 56 → 100 / 100 / 76, 85.3 → 92.0 average (+6.7 pp), all lift concentrated on stack-cube [Table V]. Qualitative analysis (§IV-D) traces recoveries from the “arm nearly frozen” no-ops regime — FailSafe-VLM detects it early and nudges the end-effector back toward the ground-truth trajectory, after which the base VLA reclaims control and finishes the task.

Why it’s interesting

Section titled “Why it’s interesting”

FailSafe sits at a productive intersection of three threads this wiki actively tracks, and it fills a specific gap in each. On the VLA Models recipe-lever board, it is a fourth entry in the “wrap frozen VLA in a companion model” pattern alongside Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents (memory-augmented LLM planner exposing π0.5 as a narrow primitive), Learning What to Say to Your VLA: Mostly Harmless VLA Steering (LFP: frozen VLA + test-time language-rewrite steering), and WAM-TTT: Steering World-Action Models by Watching Human Play at Test Time (test-time-training on the WFM’s own memory) — but is the first filed instance to steer specifically on failure states, with a fixed 10-step polling schedule instead of a language-triggered orchestrator or gated intervention. On the Synthetic Training Data axis it is the third position on the failure-data-scarcity map that EgoRecovery: Acquiring Failure Recovery Ability Through Human Recovery Demonstration (egocentric human capture) and OopsieData — a multi-lab dataset of real robot manipulation failures (multi-lab federated real-robot pooling) already stake out — FailSafe is the pure-simulator-synthesis pole, with an explicit motion-planner-based failure taxonomy and a verification-replay gate rather than filtering natural failures. It is the direct methodological sibling of DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulation, which uses simulator-synthesized failure trajectories to train a dense reward model rather than a recovery-action model — same data-generation shape, orthogonal downstream head. Finally on VLM-as-Evaluator it advances the pattern beyond the “detect and label” stage: FailSafe-VLM outputs an executable 7-DoF delta, not just a success flag (Instance Labs — Verifying Robot Learning Episode Success) or a dense per-frame reward (DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulation) — the first filed VLM-as-evaluator that closes the loop into control. The load-bearing move that makes it work is systematic verification: unlike text-only pipelines (AHA / RoboFAC) whose “recovery instructions” are ambiguous about magnitude, every FailSafe delta is guaranteed by replay to actually complete the task. Ishaan’s Slack framing (“generate failure and recovery data in sim”) is a precise summary of the pipeline’s contribution — the pattern generalizes to any manipulation task in any motion-planned simulator, so the interesting downstream question is how far the failure taxonomy (translation / rotation / no-ops) covers the failure modes that actually dominate at Luma-scale deployment.

See also

Section titled “See also”