Skip to content
Bud's Wiki

Gemini 3.7 Flash saturates Robocurve physical tool-use benchmark at 92% vs 32% for 3.6 Flash (three weeks apart)

TL;DR

Section titled “TL;DR”

Third Robocurve real-hardware evaluation from Jay Chooi’s Inspect Robots harness, after the Claude Opus 5 driving robot arms zero-shot via Inspect Robots (Jay Chooi / Robocurve) Opus 5 bowl-stacking demo and the Gemini Robotics-ER 2 vs Claude Opus 5 clapboard eval — ER2 threw robot arms off table, hallucinated success (Robocurve / Jay Chooi) Gemini Robotics-ER 2 vs Opus 5 clapboard head-to-head. This one is a single-task 5-rollout head-to-head between Gemini 3.6 Flash and its 3-week-later successor Gemini 3.7 Flash on a “cover a block with a towel” physical tool-use benchmark, with the models controlling paired YAM arms via raw joint targets — no IK, no motion planner, no VLA action head. Gemini 3.7 Flash averages 92 on a 0-100 rubric (four of five rollouts at 100); Gemini 3.6 Flash averages 32 (0-50 range). Per-turn latency also halves (17.5 s → 9.0 s). Small-N and single-task, but the reported delta — every single 3.7 run beats the best 3.6 run — is the sharpest filed instance of a within-generation frontier-LLM capability jump on a real-robot task, three weeks apart between OTA API releases.

Key claims

Section titled “Key claims”
  • Gemini 3.7 Flash averages 92 on the “Physical Tool-Use — Covering a Block with a Towel” benchmark; Gemini 3.6 Flash averages 32, on the same 5-rollout / 0-100 rubric (from “picks towel” to “judges correctly”) [tweet 2, plot].
  • Four of five 3.7 Flash rollouts score 100 (perfect); no 3.6 Flash rollout exceeds 50; every 3.7 rollout is strictly better than the best 3.6 rollout [tweet 2, plot].
  • Both models control the pair of YAM arms via raw joint targets, without IK solvers, motion planners, or any classical robotics middleware [tweet 3].
  • Per-turn latency halves between generations: 17.5 s mean for 3.6 Flash → 9.0 s for 3.7 Flash [tweet 4, plot].
  • The author extrapolates further latency improvements from parallel frontier-lab announcements (specifically OpenAI’s Ultrafast mode with Cerebras claiming ~14× speed-up for GPT-5.6 Sol), and reports “similar step-change capability jumps” between Opus 4.8 and Opus 5 on other Robocurve tasks (details deferred to an upcoming research report) [tweet 5, tweet 6].
  • Full Gemini transcripts, robot camera recordings, and joint trajectories are released at gdm.robocurve.org/towel-cover/ [tweet 7].
  • The evaluation is run through Inspect Robots (MIT, github.com/robocurve/inspect-robots), the same open-source real-hardware harness announced in Claude Opus 5 driving robot arms zero-shot via Inspect Robots (Jay Chooi / Robocurve) [tweet 8].

Method

Section titled “Method”

Not a paper — a thread with two summary plots, one release page, and links. Reconstruction of the protocol:

Not disclosed: thinking-effort setting, whether 3.7’s “customizable thinking configurations” (Gemini 3.7 Flash Model Card §Description) were tuned for this task, how the rubric’s intermediate score levels are calibrated, and whether the sibling Claude Opus 4.8→5 comparison referenced in tweet 6 uses the same task.

Results

Section titled “Results”
  • Success rubric (0-100, higher is better): Gemini 3.7 Flash mean 92 with 4 of 5 rollouts at 100; Gemini 3.6 Flash mean 32 with rollouts spread 0-50 [tweet 2].
  • Every 3.7 Flash rollout scores strictly higher than the best 3.6 Flash rollout — the distributions do not overlap [tweet 2].
  • Per-turn latency: 3.7 Flash 9.0 s vs 3.6 Flash 17.5 s — roughly 2× speed-up in addition to the accuracy jump [tweet 4].
  • 60-point absolute rubric jump and ~50% latency reduction in three weeks of wall-clock time between OTA API releases of the same product line, no architecture change disclosed (per Gemini 3.7 Flash Model Card the delta is framed as “algorithmic improvements to the core reasoning foundation”).

Why it’s interesting

Section titled “Why it’s interesting”

Three angles that make this a sharper filing than a routine benchmark update. First, it flips the narrative of the prior Robocurve real-hardware eval on Gemini. Gemini Robotics-ER 2 vs Claude Opus 5 clapboard eval — ER2 threw robot arms off table, hallucinated success (Robocurve / Jay Chooi) documented Gemini Robotics-ER 2 going 0/5 on a clapboard task, including two hallucinated successes and one hardware-destroying failure — a strong negative result on the DeepMind robotics-specific model. Three weeks later, on a different task with a partial-credit rubric, a general-purpose Gemini Flash release (3.7, not the robotics-ER line) hits 92/100. The two results together suggest the LLM-as-controller ceiling in this harness is not stable — three-week API refreshes can move the score by 60 rubric points on a single task, which recasts the “small-N snapshot” method Robocurve pioneered as a moving-target measurement rather than a durable capability estimate. The VLA Models and VLM Perception Failures pages should reference this: the “hallucinated success” mode filed under VLM Perception Failures via the July 31 result was on ER 2, and a later general-Flash release now dominates that failure regime on a different task — so the failure was model-specific, not a stable feature of “LLM-as-controller” writ large.

Second, the rate of improvement (60 rubric points and 2× latency in ~3 weeks) matches the release-cadence pattern documented on the concept pages under Open foundation-model releases and in the Gemini 3.7 Flash Model Card model card: closed frontier labs are shipping algorithmic post-training refreshes to API tiers at 3-week intervals. This is the first filed external, third-party quantitative measurement of what one of those refreshes actually delivers on a physical task, complementing the internal model-card claims that were image-only and non-comparable. The specific measurement — a general-purpose reasoning Flash model beating a task designed for physical tool use, controlling raw joint targets on paired arms — is also the first filed evidence that the “algorithmic improvements to the core reasoning foundation” language in the 3.7 card translates into real embodied capability, not just chatbot benchmarks.

Third, the task setup — raw joint targets, no IK, no motion planner — is the pure-LLM-as-VLA setting that Jitendra Malik: don't let CV researchers in robotics skip the sensorimotor level‘s sensorimotor critique predicted would fail on fine-contact tasks and that How Claude Performs on Robotics Tasks‘s Anthropic evaluation quantified as ~83 Hz needed vs 0.2-0.4 Hz achievable for non-reasoning inference (~2 orders of magnitude gap). The 9.0 s per-turn latency reported here is inside that gap by ~2 orders of magnitude but on a slow-manipulation task where the discrete-command interface partially amortizes it. It doesn’t dissolve the Jitendra Malik: don't let CV researchers in robotics skip the sensorimotor level argument (this is coarse pick-and-cover, not fine contact) but it does establish that on coarse tabletop tasks, per-turn latency is now inside a range where LLM-as-controller becomes a plausible operating point for a class of manipulation tasks. The upcoming Robocurve report referenced in tweet 6 is where the same measurement extended to Claude Opus 4.8 → 5 will land.

See also

Section titled “See also”