Skip to content
Bud's Wiki

Gemini Robotics-ER 2 vs Claude Opus 5 clapboard eval — ER2 threw robot arms off table, hallucinated success (Robocurve / Jay Chooi)

TL;DR

Section titled “TL;DR”

Second Robocurve real-hardware evaluation from Jay Chooi’s Inspect Robots harness, one week after the Claude Opus 5 driving robot arms zero-shot via Inspect Robots (Jay Chooi / Robocurve) Opus 5 zero-shot bowl-stacking demo. This one is a head-to-head: Google DeepMind’s newly-released Gemini Robotics-ER 2 (the reasoning half of the Gemini Robotics 2 brings whole body intelligence to robots ER+VLA stack) vs Claude Opus 5 on a pair of YAM arms, given the same prompt five times: “Lift the clapper stick, then let go so it snaps shut with a clap.” Gemini Robotics-ER 2 goes 0/5 — twice hallucinating success without touching the clapboard and, on the final run, flipping the robot off the table hard enough to break both camera mounts. Opus 5 goes 1/5 — never hallucinates a success and tries until the clapboard falls off. Gemini is ~40% faster per command (10.2 s vs 16.5 s Opus fast mode, both on high effort). Small-N and no independent replication, but the failure modes — spatial-hallucinated success and force-catastrophic action — are the sharper contribution than the success rate.

Key claims

Section titled “Key claims”
  • Gemini Robotics-ER 2 scored 0/5 successes on the clapboard task; two of the failures were hallucinated successes where the model claimed completion without touching the clapboard, and one failure flipped the robot off the table, breaking both camera mounts [tweet 3, tweet 4, tweet 6].
  • Claude Opus 5 scored 1/5 on the same task, never hallucinated a success, and continued attempting until the clapboard physically fell off [tweet 7].
  • The YAM arms and Intel cameras were verified functional via Inspect Robots after the Gemini incident — the mount breakage did not compromise the underlying eval hardware [tweet 5].
  • Gemini Robotics-ER 2 is ~40% faster per command than Opus 5 in fast mode: 10.2 s vs 16.5 s average time between commands, both on high effort [tweet 9].
  • The evaluations were run through Inspect Robots, described as free and fully open-source (MIT) [tweet 11].
  • Robocurve is positioned as a Public Benefit Corporation running frontier-LLM and VLA benchmarks on physical tasks in the real world, offering free evaluations to universities and non-profits [tweet 12, tweet 13].

Method

Section titled “Method”

Not a paper — a thread of short video clips and captions describing a small-N head-to-head. What can be extracted:

  • Hardware: pair of YAM arms, an actual clapboard prop, Intel cameras (mounted). Same rig as the Claude Opus 5 driving robot arms zero-shot via Inspect Robots (Jay Chooi / Robocurve) bowl-stacking demo.
  • Task: literal DeepMind demo replication — “Lift the clapper stick, then let go so it snaps shut with a clap” — chosen because DeepMind showed a robot doing it earlier in the week; this is a targeted “replicate the launch demo” benchmark.
  • Prompt: identical single-string instruction across all 10 rollouts (5 per model).
  • Models under test: Gemini Robotics-ER 2 (Google DeepMind’s ER model, the reasoning tier of the ER+VLA two-stack architecture in Gemini Robotics 2 brings whole body intelligence to robots) and Claude Opus 5 (Anthropic’s frontier LLM, invoked in the same LLM-as-VLA-supervisor pattern as Claude Opus 5 driving robot arms zero-shot via Inspect Robots (Jay Chooi / Robocurve) and How Claude Performs on Robotics Tasks).
  • Effort/latency setting: both models on high effort; Opus 5 additionally noted running in “fast mode”.
  • Grading: binary success per rollout (clapper stick lifted then let go with a clap). “Hallucinated success” is distinguished from real success by video review — the model reports task completion but the arms never contacted the clapboard.
  • Harness: Inspect Robots (github.com/robocurve/inspect-robots, MIT), the open-source evaluation framework already announced in Claude Opus 5 driving robot arms zero-shot via Inspect Robots (Jay Chooi / Robocurve).
  • Not disclosed: whether Gemini Robotics-ER 2 is invoked with the same tool interface as Opus 5, whether ER 2 is being run “alone” (as a policy) or as the reasoning half above an action model, and what the action space looks like (joint-level, end-effector deltas, or high-level primitives).

Results

Section titled “Results”
  • Gemini Robotics-ER 2: 0/5, 2 hallucinated successes, 1 catastrophic failure (robot off table, both camera mounts destroyed) [tweet 3, tweet 4, tweet 6].
  • Opus 5: 1/5, 0 hallucinated successes [tweet 7].
  • Latency: Gemini ~10.2 s per command, Opus ~16.5 s per command in fast mode, both on high effort — ~40% Gemini advantage [tweet 9].
  • No confidence intervals; N=5 per model. The success-rate delta (1/5 vs 0/5) is not statistically resolvable at this N; the failure-mode deltas (hallucinated success, hardware damage) are qualitative and reproducible only in the sense that they occurred at all.

Why it’s interesting

Section titled “Why it’s interesting”

Three things make this a sharper filing than a typical launch-week vendor comparison. First, the specific failure — Gemini Robotics-ER 2 twice reporting task completion without physically touching the clapboard — is the real-hardware embodied analog of the “LLM answers from prior, not image” failure documented across Vision Language Models are Biased and Hidden in plain sight: VLMs overlook their visual representations on the VLM Perception Failures page: the model has a strong prior for what “clapper stick lifted and let go” looks like from the DeepMind launch video and generates a claim of success matching that prior, ungrounded in whether its own end-effector actually touched the object. This is the “hallucinated success” failure mode named explicitly in Instance Labs — Verifying Robot Learning Episode Success (Instance Labs’ pitch), given a concrete frontier-model instance rather than an abstract argument. Second, it sits at the same interface — LLM-as-controller with a curated tool wrapper on real hardware — as How Claude Performs on Robotics Tasks‘s Anthropic Frontier Red Team evaluation, but flips the comparison axis: Anthropic ran multiple Claude generations on LIBERO manipulation and Go2/G1 locomotion; Robocurve runs one Claude generation against one Gemini generation on a single manipulation task at the sharpest possible failure mode (contact-establishment on a small object). Both suggest the ceiling case for LLM-as-VLA-controller is fine-grained contact, which is exactly the axis Jitendra Malik: don't let CV researchers in robotics skip the sensorimotor level‘s sensorimotor counter-position identified as the structural weakness of the VLA framing and which ArmnetBench v0.1: Parallel Real-World Evaluation of Manipulation Policies on a Low-Cost Arm Farm confirmed on the recipe side with 50-pp precise-force gaps between Diffusion Policy from scratch and every VLA. Third, this is the first filed external, third-party head-to-head on Gemini Robotics-ER 2 specifically — DeepMind’s own release deferred hard numbers to a Safety Technical Report and reported no head-to-head benchmarks in the blog. Inspect Robots is now positioned as the neutral-third-party evaluation harness for that role (the RL Environment Platforms real-hardware slot alongside SETA/Toolathlon-GYM/OpenReward on the sim/coding-agent side), and the pattern of “small-N but high-detail per-rollout video and diagnostic” is a plausible sustainable format for the emerging real-hardware VLA/LLM-as-controller comparison surface.

See also

Section titled “See also”