Skip to content
Bud's Wiki

How Claude Performs on Robotics Tasks

TL;DR

Section titled “TL;DR”

Anthropic’s Frontier Red Team evaluates several Claude generations (Opus 4 → Opus 4.7 + Mythos Preview) alongside GPT-5.4 and Gemini 3.1 as controllers for physical robots across three regimes — classic control (pendulum, hopper, TwinFlipper pinball), locomotion on a Unitree G1 humanoid and Go2 quadruped, and manipulation with a 7-DoF Franka Panda arm on LIBERO tasks — under four control interfaces: direct low-level torque commands, programmatic Python controllers, high-level control of pretrained policies / VLAs, and end-to-end RL supervision. The headline finding is that capability depends far more on interface than on model strength: language models cannot reliably drive joints themselves (no model successfully stands up the G1 humanoid), but they perform substantially better when supervising a pretrained gait or a VLA, and small orientation aids (a compass tool for locomotion, a cursor tool for manipulation) close far more of the gap than richer visual inputs (depth heatmaps, segmentation overlays, third-person cameras). The report also introduces TwinFlipper as a novel chaotic-dynamics classic-control benchmark not seen in pretraining.

Key claims

Section titled “Key claims”
  • Direct low-level control fails on high-DoF platforms: no tested model successfully stands the 29-DoF Unitree G1 humanoid from a collapsed pose, and real-time control would require ~83 Hz vs current non-reasoning inference at 0.2–0.4 Hz — roughly two orders of magnitude gap [§Locomotion].
  • Programmatic control (LLM writes a Python controller) substantially outperforms direct control on the 12-DoF Go2 quadruped, with Opus 4.6/4.7 and Mythos Preview balancing the robot for nearly two seconds with torque-force control [§Locomotion].
  • Later Claude generations improve mainly through better adaptation across attempts, not first-try performance: on Pendulum and TwinFlipper, first-try scores are similar across models, but subsequent attempts show much larger gains for newer models [§Classic control].
  • Pretrained-policy supervision dramatically expands manipulation capability: pairing the LLM with the MolmoAct VLA on LIBERO increases task success and progress for every tested model, but every LLM-guided VLA still performs worse than MolmoAct running alone [§High-level manipulation].
  • Newer Claude models (Opus 4.5/4.6/4.7) defer to the VLA less on novel LIBERO-like tasks where the VLA fails, and Opus 4.5/4.6 and Gemini 3.1 outperform MolmoAct alone on those unseen tasks — they are “better at recognizing when the policy is failing, even if they are not yet able to correct those failures directly” [§Novel manipulation tasks].
  • Simple orientation tools beat richer visual inputs: a compass (facing direction in degrees) uplifts every locomotion configuration tested, and a cursor tool that annotates gripper position raises Mythos Preview’s manipulation success from 6% → 32% on a 10-task subset — while depth heatmaps, segmentation overlays, and third-person cameras are roughly neutral for most models [§Perception aids].
  • Visual perception is a stronger limitation for older models: replacing images with Gemini-3.1-generated text descriptions improves older Claude performance but hurts Opus 4.6/4.7, evidence newer models extract more spatial information from raw pixels [§Perception aids].
  • Extra reasoning budget frequently hurts, not helps: on classic-control tasks, newer models regress with higher reasoning budgets (interpreted as overengineering simple problems), and on manipulation, reasoning has no major effect on Claude-family models while it hurts Gemini 3.1 and GPT-5.4 [§Reasoning].
  • On real-hardware navigation with the physical Go2, an egocentric crosshair overlay both helps (aligning down hallways) and hurts (Opus 4.6 walked into a trash can it declared “out of the way” because it was left of the crosshair) — a concrete failure mode of purely visual aids without geometric grounding [§Real-world].
  • On RL supervision (model trains its own policy given a scaffold + GPU), GPT-5.4 and Mythos Preview consistently produce the most competent policies; Mythos Preview is the only model that consistently learns TwinFlipper end-to-end [§Classic control, §RL supervision].

Method

Section titled “Method”

The evaluation is factored into three task regimes × four control interfaces. Direct control: the model outputs low-level commands (torques, forces, or 7-DoF end-effector deltas) each step in a paused Mujoco simulator — pausing between LLM calls approximates an upper bound on capability by removing real-time latency. Programmatic control: the model writes a Python controller that maps observations to actions during execution. Policy control: the model issues high-level commands (velocity for the Go2 joystick policy, or accept/edit/replace of proposed actions from MolmoAct on LIBERO). Reinforcement learning supervision: the model is given a training scaffold, GPU, and visualization environment, and controls the reward function, environment, and architecture over a ~4-hour training run. Classic control uses inverse pendulum, hopper, and TwinFlipper (novel pinball task, cumulative airtime above a threshold). Locomotion uses the 12-DoF Go2 quadruped and 29-DoF G1 humanoid on stand-up and balance tasks. Manipulation uses LIBERO kitchen scenes on a Franka Panda. Perception aids tested include a compass tool (yaw in degrees), a cursor tool (annotated point on gripper cam with object identity + distance), a depth heatmap overlay, a semantic-segmentation overlay, a third-person chase camera, and text-only substitution via Gemini 3.1 VQA.

Results

Section titled “Results”

Across classic control, Opus 4.6 and 4.5 outperform earlier generations on almost all tasks except TwinFlipper-direct (all models poor) and hopper-velocity (noisiest task); gains concentrate in later attempts rather than first-try scores. On the Go2 quadruped, direct low-level balancing is hard for every model, but with programmatic Python controllers Opus 4.6/4.7 and Mythos Preview keep the robot balanced for nearly two seconds. On the G1 humanoid, no model successfully stands it up from collapsed; Opus 4 → 4.7 show measurable but small progress on maintaining balance from an already-upright pose. On direct manipulation (LIBERO with 7-DoF end-effector commands), Opus 4.6 substantially raises the rate of guiding the arm to the target, contacting it, and grasping it vs Opus 4/4.1, though full end-to-end task success remains rare. On policy-supervised manipulation with MolmoAct, every LLM-guided pairing improves over direct control, but every configuration underperforms MolmoAct running alone — Mythos Preview is worst here because it overrides the VLA more often than warranted. On the eleven-task high-level Go2 navigation suite, Opus 4.7→Mythos Preview delivers the biggest single-generation jump; the third-person chase camera helps Opus 4.7 (+5.8) and Mythos Preview (+10.7) but is neutral-to-negative for Opus 4.6 and earlier. The compass tool uplifts every configuration; the cursor tool raises Mythos Preview manipulation success from 6% → 32% on a 10-task subset. Real-hardware Go2 tests validate both encouraging cases (crosshair-guided hallway alignment) and concrete failure modes (obstacle occlusion + false-safe reasoning about crosshair position).

Why it’s interesting

Section titled “Why it’s interesting”

This is the sharpest filed evaluation to date of frontier general-purpose LLMs as robot controllers, and it lands squarely on the design axis this wiki has been tracking. The results directly reinforce the VLA Models concept’s tool-use / physical-orchestrator framing from VoLo: A Physical Orchestrator for Open-Vocabulary Long-Horizon Manipulation and Learning What to Say to Your VLA: Mostly Harmless Vision Language Action Model Steering: models are much more useful supervising pretrained policies than driving joints, and the newer Claude generations are getting better at recognizing when the underlying VLA is failing on novel tasks — exactly the “black-box VLA as callable tool” surface those papers formalized. It complements Hy-Embodied-0.5-VLA: From Vision-Language-Action Models to a Real-World Robot Learning Stack‘s and π*0.6: a VLA That Learns From Experience (RECAP)‘s bet on action-pretraining + RL by measuring what capability the pretrained-VLA-plus-LLM-supervisor stack contributes on top. The compass/cursor finding is a concrete answer to the perception debate in VLM Perception Failures: telling models where they’re facing beats showing them richer pictures. Finally, this is the first filed frontier-lab report treating “LLM-driven robotics” as a safety evaluation axis with capability trending upward per generation, which makes it a useful reference point for the RL Environment Platforms evaluation surface.

See also

Section titled “See also”