Skip to content
Bud's Wiki

Spotter: Let the Embodied Model Lead, and the VLM Reflect for It

TL;DR

Section titled “TL;DR”

Spotter inverts the VLM-led robot-agent recipe: the embodied policy (Cosmos Policy or π₀.₅) stays in charge and executes action chunks continuously, while a locally-served Qwen screener runs in parallel one chunk behind, and a frontier VLM judge (Qwen or GPT-6 Astra) is called only after two consecutive flagged chunks confirm an error. On confirmation the judge takes over with a short motion-primitive plan, states what the next supervised window should show, and hands back to the policy. The paper first measures why current embodied models cannot self-repair (fewer than 1 in 10 failures corrected across three protocols; post-failure scorers collapse toward random even when they discriminate at normal states), then shows Spotter lifts RoboCasa success by up to 5.6 pp (Cosmos Policy) and 7.5 pp (π₀.₅) with GPT and 1-shot, with successful episodes taking only 13 s / 16 s longer than the policy alone and ~70% less time than Harness VLA at matched VLM — no policy training, no scene resets at test time, and 53% → 83% on a real Franka Research 3 (long-horizon task 30% → 70%).

Key claims

Section titled “Key claims”
  • Current embodied models cannot repair their own errors. Across resampling, language feedback, and recovery fine-tuning, fewer than 10% of oracle-detected failures are corrected within 3 attempts; recovery fine-tuning helps little because <3% of ordinary rollouts yield usable recovery data [§3.1, Fig. 2a].
  • Test-time scorers lose discrimination exactly where it matters. Cosmos Policy’s value head and Consistency-Consensus discriminate successful vs. failed candidates well at normal states but drop to near-random after a failure; a VLM judge given only current images is similarly unable to rank successful recovery candidates (53% / 51% pairwise accuracy with GPT-6 Astra / Qwen at post-failure states) [§3.1, Fig. 2b].
  • The remedy must come from a VLM reading the episode history and text, not from the embodied model. Judging a repair from a repeated failure requires access to the failed attempt, prior context, and robot telemetry — a frontier VLM composes a repair over motion primitives and verifies it in the next supervised window [§3.1, §3.2].
  • Parallel supervision with a lead limit replaces the VLM-led critical path. The policy runs at most K=2–4 chunks ahead of the latest screened chunk (Eq. 1); apart from confirmed errors this bound is the only reason the policy ever pauses [§3.2].
  • Two-flag gating keeps the frontier VLM off the hot path. A locally served Qwen screener (recall-biased threshold, screener failures count as flags) only escalates to the judge after two consecutive flagged chunks or an empty-gripper rule [§3.2, Eq. 2, Appendix B].
  • The judge repairs via a small primitive vocabulary plus an expectation. The judge emits a short plan over 9 primitives (Move to point, Move by offset, Lift, Rotate wrist, Set gripper, Retreat, Move both arms, Act, Hand back) and must state an observable expectation; the next supervised window is observation-only and must verify the expectation before any further repair, and a failed expectation forces a different kind of repair rather than a retried offset [§3.2, Table 2].
  • No scene resets at test time; only pose rollback. Spotter records the arm pose at each chunk boundary and uses the Retreat primitive to return to a recorded earlier pose, which keeps the recipe real-robot-deployable (Section 3.2 “Return the arm, not the scene”). Simulator resets are used only during offline lesson learning.
  • RoboCasa success rates (24 tasks, 1,200 episodes). Spotter (GPT, 1-shot) hits 73.5% on Cosmos Policy and 71.8% on π₀.₅ vs. embodied-model-at-step-limit baselines of 67.9% and 64.3% — +5.6 pp and +7.5 pp, concentrated on pick-and-place (+10.3 pp and +9.0 pp) [Table 4].
  • Spotter with Qwen matches privileged Harness VLA. With no privileged info and the same local Qwen, Spotter 1-shot reaches 71.8 / 70.4, exceeding Harness VLA’s privileged-info numbers (71.3 / 69.1) and sharply beating Harness VLA without privilege (65.2 / 60.5), which falls below the embodied model alone on non-PnP tasks [Table 4].
  • The screener earns its keep. Removing it (judge examines every window) adds ~2 pp (Cosmos 70.0 → 72.0, π₀.₅ 68.8 → 70.8) but drives the frontier VLM onto the common path [Table 4, “Spotter (GPT) no screener”].
  • Latency: successful episodes stay close to the policy alone. With Qwen, judge is consulted <1× per successful episode; successful episodes add only 13 s (Cosmos Policy) and 16 s (π₀.₅) over the embodied model alone — ~70% less wall time than Harness VLA at matched VLM [§4.4, Fig. 6, Table 5].
  • Bimanual Hard setting of RoboTwin 2.0 widens the Qwen↔GPT gap. From π₀.₅ 47.2%, Spotter reaches 51.2% (Qwen) and 57.0% (GPT) — a 5.8 pp Qwen–GPT gap vs 1.7 / 1.4 pp on RoboCasa [§4.2, Fig. 4b].
  • Real Franka Research 3 (3 tasks, π₀.₅ fine-tuned on teleop). Spotter (GPT, 1-shot) raises average success from 53% (16/30) to 83% (25/30); the long-horizon chained-subtask variant moves 30% → 70% [§4.3, Fig. 5].
  • Lesson library + example bank are the only learned artifacts. 77 general lessons distilled from a held-out seed (195) live in memory_bank/global/ with evidence episodes + confidence + falsification criteria; a 151 MB example bank adds one matching worked example to the judge’s prompt per episode in the 1-shot setting [project page §“Lesson library”].

Method

Section titled “Method”

Spotter structures a frozen embodied policy π_θ, a locally-served Qwen screener S, and a frontier VLM judge J around a lead-limited parallel loop (Fig. 3c). The policy executes action chunks C_i back-to-back with no checkpoint between them; each finished chunk is streamed to the screener in parallel. A lead limit i − k_latest ≤ K (K = 2–4) is the only hard synchronization point — if the policy is K chunks ahead of the latest check, it stops at the chunk boundary until S catches up. Because S runs slower than π_θ, each screening call batches all chunks finished since the previous one, not only the latest. S returns a risk score r_i ∈ {0, 1, ⊥}; the threshold is low (recall-biased) and screener failures ⊥ count as flags.

The judge J is called under two conditions: (i) two consecutive flagged chunks, or (ii) an immediate rule for empty-gripper evidence. When called, J sees the episode history, current observations, and recent chunks, and returns a verdict v ∈ {ok, ✗}, a free-text diagnosis, and — on intervention — an expectation plus a short plan over 9 primitives (Table 2: Move to point, Move by offset, Lift, Rotate wrist, Set gripper, Retreat, Move both arms, Act, Hand back). The executor lifts pixel targets in named camera views to 3D using depth maps (J itself never sees depth). After each segment J gets a fresh image and either continues the plan, declares the problem fixed, or gives up; π_θ then resumes from the repaired state with no rollback.

The verification mechanic is load-bearing: after a repair the next supervised window is observation-only and the judge must first say whether the expectation held. If it did, J must not repeat the repair; if it didn’t, the next intervention must try a different kind of repair, not a new offset. The paper flags that without this discipline J tends to “repair and let go, unaware of whether it worked.” Three failure cases are categorized explicitly: no damage (undo by returning the arm to an earlier recorded pose), repairable damage (relocate and complete), and irreversible damage (unrecoverable and not Spotter’s claim — the paper notes paradigms (a) and (b) can’t recover either).

Spotter never trains π_θ. The learned artifacts are two frozen stores: (i) a lesson library of 77 general lessons distilled from a held-out seed (195) — examples: “A closed gripper is not a grasp; only the object leaving its support proves one”; “Repair the moment of the missed grasp, not the retreated pose the policy left you in”; “Three failed repairs on one trajectory means stop and let the policy run out” — each with evidence episodes, confidence, and a falsification criterion; and (ii) a 1-shot example bank (151 MB) with the frames the judge saw + verdict + successful repair for both Cosmos Policy and π₀.₅. One matching example is added to the judge’s prompt per episode in the 1-shot setting.

Results

Section titled “Results”
  • RoboCasa, 24 tasks × 50 episodes = 1,200 episodes: Cosmos Policy baseline 67.9% at 1.8× step limit → Spotter (GPT, 1-shot) 73.5% (+5.6 pp), PnP sub-split 53.2 → 65.5 (+10.3 pp). π₀.₅ baseline 64.3 → 71.8 (+7.5 pp), PnP 56.8 → 67.0 (+9.0 pp) [Table 4].
  • Spotter (Qwen, local, no privileged info) beats privileged Harness VLA. 71.8 / 70.4 vs 71.3 / 69.1 at same budget; Harness VLA without privilege collapses to 65.2 / 60.5, falling below the embodied model on mechanism-operation tasks (−9.2 / −11.7 pp) [Table 4].
  • No-screener ablation: Spotter (GPT) 72.0 / 70.8 vs with-screener 70.0 / 68.8 — the screener costs ~2 pp but keeps the frontier VLM off the common path [Table 4].
  • Detecting a failure is not enough. Fixed-retry with the oracle detector reaches only 63.1 / 67.1 on π₀.₅ / Cosmos — same tier as the embodied model alone, consistent with §3.1’s finding that current policies cannot act on knowledge of their own failure [Table 4].
  • RoboTwin 2.0 Hard (bimanual). π₀.₅ 47.2% → Spotter 51.2% (Qwen) / 57.0% (GPT) — the GPT–Qwen gap widens to 5.8 pp from ~1.5 pp on RoboCasa, matching the paper’s claim that longer-horizon bimanual tasks lean more on the frontier VLM [§4.2, Fig. 4b].
  • Real Franka Research 3, 3 tasks, π₀.₅ fine-tuned on teleop. Average success 53% (16/30) → 83% (25/30) with Spotter (GPT, 1-shot). Long-horizon chained-subtask variant 30% → 70% — the regime where one error in either subtask kills the whole episode [§4.3, Fig. 5].
  • Latency on successful episodes with Qwen: +13 s (Cosmos Policy) / +16 s (π₀.₅) over the embodied model alone — ~70% less than Harness VLA at matched VLM; the small screener hides 36–38 s / 29–30 s of supervision per episode (~60–80% of the sync-serial cost) because execution continues while chunks are checked [§4.4, Table 6].
  • Judge-call accounting (successful / failed episodes): Cosmos + Spotter (Qwen) full-context 0.6 / 10.9 judge calls, 0.0 / 3.7 repairs — the common path is screener-dominated, judge only enters on actual failure [Table 5].

Why it’s interesting

Section titled “Why it’s interesting”

Spotter is the sharpest filed instance of the parallel-supervision cell of the “wrap frozen VLA in an orchestrator” row of the VLA Models recipe-lever board — distinct in a specific, measurable way from the three existing filed instances. Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents Harness VLA runs the LLM planner at every subtask boundary (VLM-led, serial, requires privileged success-signal access to work with Qwen); FailSafe: Reasoning and Recovery from Failures in Vision-Language-Action Models FailSafe polls a companion VLM every 10 steps and emits a 7-DoF delta (fixed schedule, no two-flag gating); and ARC: A Reasoning Recipe for Robot Foundation Models ARC runs an async background VLM at ~1 Hz and fine-tunes the policy to consume the trace. Spotter’s load-bearing structural moves are (i) the K-chunk lead limit as the only hard synchronization point other than confirmed errors, (ii) two-flag gating with recall-biased screening, and (iii) the expectation-verification mechanic that forces the next intervention after a failed expectation to try a different kind of repair — which the paper identifies as the discipline that keeps the judge from “repair-and-let-go” behavior. The headline measured consequence is that Spotter (Qwen, no privileged info) beats Harness VLA (Qwen, with privileged success-signal access) by 0.5–1.3 pp on both backbones, and the privilege-stripped Harness VLA falls below the embodied model alone — the clearest filed evidence that where the VLM sits in the loop matters more than which VLM you plug in, especially for Qwen-tier local models.

The §3.1 diagnostic is the second contribution and sharpens the long-running debate on the recipe-lever board. It is a direct quantitative follow-on to Behavioral Cloning Mystery Behavioral Cloning Mystery (same “diagnose BC pathologies at the objective level” shape, with perception ruled out as a confounder) and gives a concrete number — <10% repair rate across three protocols, scorers collapsing to near-random after a failure — to the sensorimotor-counter-position’s prediction (Jitendra Malik: don't let CV researchers in robotics skip the sensorimotor level) that contact-establishment failures are the regime where pure VLA scaling has not delivered. The scorer-collapse result is also the sharpest critique on file of the inference-time-scaling branch of the board (Cosmos Policy’s value head, Consistency-Consensus / Consistency-Exploring, GeoBoN, RCS): the scorers these methods depend on lose their skill exactly where they are most needed, which recasts the recovery-fine-tuning branch (EgoRecovery: Acquiring Failure Recovery Ability Through Human Recovery Demonstration EgoRecovery, Enact’s DAgger pipeline from Enact launch — post-training infrastructure that generates targeted recovery data for robotics VLAs) as attacking the same underlying “the policy learned only from successful demos” problem Spotter sidesteps entirely by handing control to a VLM that reads the full episode history.

The learned-artifact design — 77 natural-language lessons with evidence episodes + confidence + falsification criteria, plus a 1-shot example bank keyed on frames + verdict + repair — fits the Parametric memory “external-symbolic + experiential + evolution-heavy” cell cleanly, sibling to the RobotUse: Allocating Computation, Context, and Decisions RobotUse playbook recipe and Decentralized Multi-Agent Systems with Shared Context DeLM’s verified-gist shared context. The novel disciplinary move is the falsification criterion per lesson: each stored lesson records what observation would make it wrong, which is a Dynamics-axis discipline none of the other filed experiential-memory recipes name explicitly.

See also

Section titled “See also”