Comparing Fable 5.1 to Fable 5 on robotic manipulation

TL;DR

Fourth Robocurve real-hardware evaluation from the Inspect Robots harness, and the first one that (a) uses 20 rollouts per condition instead of 5 and (b) puts Claude Fable 5 head-to-head against its point-update successor Claude Fable 5.1 on two YAM-bimanual manipulation tasks. On “block into bowl” Fable 5.1 completes 8 of 20 (40%) vs Fable 5’s 1 of 20 (5%) — an 8× success ratio at ~1.5× fewer output tokens per run (12.9k vs 19.2k). On the harder “puzzle piece into groove” task Fable 5.1 completes 2 of 20 (10%) vs Fable 5’s 0 of 20, at 10.5k vs 16.3k tokens. Unlike the prior Gemini 3.7 Flash saturates Robocurve physical tool-use benchmark at 92% vs 32% for 3.6 Flash (three weeks apart) Gemini 3.6→3.7 measurement, both models here write end-effector poses and an IK solver converts to joint commands — the shallower LLM-as-controller interface. Full transcripts, MP4 videos, and Rerun .rrd trajectories for all 80 runs are released.

Key claims

Fable 5.1 completes the block-into-bowl task in 8 of 20 trials (40%) vs Fable 5’s 1 of 20 (5%) — an 8× improvement at the same 20-rollout budget [Results table].
On the harder puzzle-piece-into-groove task Fable 5.1 completes 2 of 20 (10%) vs Fable 5’s 0 of 20 (0%) [Results table].
Fable 5.1 produces fewer output tokens per run on both tasks: 12.9k vs 19.2k on block-into-bowl (1.49×), and 10.5k vs 16.3k on puzzle-into-groove (1.55×) [Results table].
Mean stage-completion (0-4 human-graded ladder) rises Fable 5→5.1 from 1.30 to 2.40 on block-into-bowl and from 1.50 to 2.35 on puzzle-into-groove [Results table].
The stage histogram shows Fable 5 stalls at first contact (stage 1) in 13 of 20 bowl trials, never closing the gripper on the block; Fable 5.1’s failure wall on the puzzle task is at final insertion (stage 3), not initial contact [Stage-histogram section].
The control interface is move_to(x, y, z, yaw, pitch, roll, gripper) per arm — the model emits absolute end-effector poses and the robot’s IK solver converts them to joint angles [Technical specifications table].
Observations are three camera views (top, left wrist, right wrist) plus proprioceptive state each turn; policy is the Inspect Robots agent policy at medium thinking effort, 20-LLM-call budget, 25% speed cap, default safety guardrails on [Technical specifications table].
Fable 5 exhausted its 20-call budget in 18 of 20 bowl trials, while Fable 5.1 usually stopped on its own — so the reported delta may partially reflect budget-limited Fable 5 rather than a pure capability gap [Limitations section].
Both models were run only at medium thinking effort; whether the gap holds at other effort levels is untested [Limitations section].
All 80 trials release full LLM transcripts, MP4 camera videos, and Rerun .rrd trajectory files for external replay and re-scoring [All runs table].

Method

Two tasks on the same bimanual I2RT YAM rig used in the Claude Opus 5 driving robot arms zero-shot via Inspect Robots (Jay Chooi / Robocurve) / Gemini Robotics-ER 2 vs Claude Opus 5 clapboard eval — ER2 threw robot arms off table, hallucinated success (Robocurve / Jay Chooi) / Gemini 3.7 Flash saturates Robocurve physical tool-use benchmark at 92% vs 32% for 3.6 Flash (three weeks apart) evals:

Task A — Block into bowl: “Pick up the red block from the table and place it inside the bowl.” 20 trials per model.
Task B — Puzzle piece into groove: “Pick up the round blue puzzle piece by the knob at its center and place it into the matching circular groove in the board.” 20 trials per model.

Every trial is scored on the highest stage it reaches (0-4): 0 no purposeful approach → 1 contact with object → 2 lifted clear of table → 3 positioned above deposit point → 4 placed in final position. This gives partial credit (unlike the earlier binary or 0-100-rubric protocols) and lets a failed run still record where it stalled.

The control interface is what makes this measurement distinct from prior Robocurve evals: the LLM emits absolute end-effector poses move_to(x, y, z, yaw, pitch, roll, gripper) per arm as tool calls, and the robot’s IK solver converts them to joint angles. This is the shallower LLM-as-controller interface — Gemini 3.7 Flash saturates Robocurve physical tool-use benchmark at 92% vs 32% for 3.6 Flash (three weeks apart) and Gemini Robotics-ER 2 vs Claude Opus 5 clapboard eval — ER2 threw robot arms off table, hallucinated success (Robocurve / Jay Chooi) evaluated the raw-joint-target interface (no IK). The blog explicitly notes both interfaces are supported: “LLMs can also drive the joints directly with no solver, as in our previous results.” Every turn the model receives three camera views (top, left wrist, right wrist) plus proprioceptive state; runs use the Inspect Robots agent policy at medium thinking effort with a 20-LLM-call budget, 25% speed cap, and default safety guardrails on.

Two models under test:

Claude Fable 5 (claude-fable-5) — the prior generation.
Claude Fable 5.1 (claude-fable-5-1) — a point update / post-training refresh.

Model identifiers are Anthropic API strings; Fable is Claude’s model line (the URL path is anthropic.robocurve.org/fable-5.1). The blog does not disclose thinking-effort variation, sampling temperature, or whether the two models share tokenizers.

Every counted trial produces a full LLM transcript (HTML), an MP4 camera video, and a Rerun .rrd trajectory file, all downloadable — one row per trial in the “All runs” table.

Results

Block into bowl: Fable 5 mean stage 1.30, 1/20 completions (5%), 19.2k output tokens/run. Fable 5.1 mean stage 2.40, 8/20 completions (40%), 12.9k tokens/run. 8× success ratio at 1.49× fewer tokens.
Puzzle into groove: Fable 5 mean stage 1.50, 0/20 completions (0%), 16.3k tokens/run. Fable 5.1 mean stage 2.35, 2/20 completions (10%), 10.5k tokens/run. Non-zero from zero at 1.55× fewer tokens.
Failure structure: Fable 5 fails at first contact 13/20 times on the bowl task (never closes gripper on block); Fable 5.1’s failures cluster at the final insertion step on the puzzle task rather than at initial contact.
Budget confound: Fable 5 hits its 20-LLM-call budget in 18/20 bowl trials, whereas Fable 5.1 usually terminates on its own. A larger call budget for Fable 5 was not tested; the 8× ratio may narrow under a call-budget-matched protocol.

Why it’s interesting

Three angles that make this a sharper filing than a routine version bump.

First, this is the first Robocurve eval that establishes an 80-rollout sample size on the same rig (20 per model per task × 2 models × 2 tasks) — 4× the per-condition N of the prior three evals (Claude Opus 5 driving robot arms zero-shot via Inspect Robots (Jay Chooi / Robocurve) 5-rollout bowl-stacking, Gemini Robotics-ER 2 vs Claude Opus 5 clapboard eval — ER2 threw robot arms off table, hallucinated success (Robocurve / Jay Chooi) 5-rollout clapboard, Gemini 3.7 Flash saturates Robocurve physical tool-use benchmark at 92% vs 32% for 3.6 Flash (three weeks apart) 5-rollout towel-cover). Combined with the schema-versioned EvalLog released for all 80 runs per Inspect Robots — An Open-Source Evaluation Framework for Physical AI (Robocurve), this addresses the “small-N snapshot” methodological ceiling that made the earlier evals a moving-target measurement rather than a durable capability estimate. The 8/20 vs 1/20 gap on block-into-bowl is now large enough for a Fisher’s exact test to reject at the standard threshold — a first for a Robocurve real-hardware Claude-generation comparison.

Second, this is the first Robocurve datapoint on the end-effector-pose + IK-solver interface rather than the raw-joint-target interface. The Gemini 3.7 Flash saturates Robocurve physical tool-use benchmark at 92% vs 32% for 3.6 Flash (three weeks apart) Gemini 3.6→3.7 measurement and the Gemini Robotics-ER 2 vs Claude Opus 5 clapboard eval — ER2 threw robot arms off table, hallucinated success (Robocurve / Jay Chooi) ER 2 vs Opus 5 comparison both used the deeper raw-joint interface, which per Jitendra Malik: don't let CV researchers in robotics skip the sensorimotor level‘s sensorimotor counter-position was expected to be the harder regime. This blog explicitly notes both interfaces are supported and both are being evaluated in parallel; giving the LLM an IK solver moves the failure surface from “can the model articulate the joint kinematics” to “can the model plan and execute end-effector poses under contact.” The failure mode Fable 5 exhibits on the bowl task (13/20 stalls at first contact, never closes gripper) is exactly a contact-establishment failure — matching the pattern ArmnetBench v0.1: Parallel Real-World Evaluation of Manipulation Policies on a Low-Cost Arm Farm and T-Rex: Tactile-Reactive Dexterous Manipulation frame as the specific hole VLA scaling has not filled, now observed for LLM-as-controller with IK.

Third, the 1.5× token reduction alongside the 8× success ratio is a data point on the token-efficiency vs capability question that Gemini 3.7 Flash saturates Robocurve physical tool-use benchmark at 92% vs 32% for 3.6 Flash (three weeks apart)‘s Gemini 3.7 latency measurement raised (17.5s → 9.0s per turn). Fable 5’s 18/20 budget-exhaustion rate on the bowl task means the token gap is confounded by call-budget saturation — but even so, Fable 5.1 is successfully completing more tasks inside the budget while Fable 5 is running out of it. The upcoming call-budget-matched follow-up would be the clean experiment. This complements the How Claude Performs on Robotics Tasks Anthropic Frontier Red Team internal evaluation (older Claude generations on LIBERO) with the same-lab external counterpart on real hardware, one model generation later, on the physically-grounded IK-solver interface.