HumanoidToolBench: Benchmarking Humanoid Tool Use from Selection to Mobile Execution

TL;DR

HumanoidToolBench is an 18-task benchmark that jointly evaluates whether a humanoid robot can (a) select a suitable tool from a cluttered table and (b) use it, with and without locomotion, across three scenarios (Move, Retrieve, Break) × three execution levels (L0 select-only, L1 stationary use, L2 mobile use) × two tool-set modes (standard vs. decoy). The companion ToolBook dataset supplies 3,003 simulation and 91 real-robot teleop demonstrations collected on a Unitree G1 with eight teleoperators + Meta Quest. Evaluating seven policies in simulation and three on real hardware exposes a sharp selection ↔ execution gap: adding locomotion collapses success relative to L1 for every policy that could do L1 at all, and GR00T N1.7 (the strongest VLA in the sweep) loses selection accuracy on unseen tools and continues to execute confidently under unrelated instructions. The authors release code + weights + data under permissive licenses.

Key claims

Mobile use is not a free extension of stationary use. For every policy with any L1 success, L2 success is strictly lower across all three scenarios in both standard and decoy modes [Results tables; “For every policy, success falls from L1 to L2 whenever L1 success is above zero”].
A WAM-shaped policy wins outright on this benchmark. FastWAM leads 6-of-9 standard-mode L0+L1+L2 cells (e.g. standard L1 Break 86, L2 Break 69; decoy L2 Break 61) and GR00T N1.7 leads 3-of-9 (standard L0 Select 77); every imitation-learning and non-WAM-VLA baseline sits ≤30 across the board [standard-mode and decoy-mode tables].
Decoy mode degrades selection and execution for the two strongest policies. Both GR00T N1.7 and FastWAM score lower in decoy mode in all three L1 scenarios than in standard mode [decoy-mode table vs. standard-mode table].
Selection accuracy drops on unseen tools even when the policy was pre-trained on them. A GR00T N1.7 probe trained on standard mode only sees correct first contact fall 85→74% on BallMove and 59→36% on BallRetrieve when evaluated on unseen tools [Analysis §Probes].
Instruction following is not load-bearing for execution. Told “The weather is nice today”, a GR00T N1.7 trained on standard mode still breaks the ice in 84% of rollouts (vs. 77% with the correct instruction) — the policy executes the training-time behavior conditional on scene rather than on language [Analysis §Probes].
The whole benchmark isolates five sub-successes along the rollout. Each episode logs candidate contact → correct first contact → correct-tool lift → tool-target contact → success, letting the gap between selection and execution be read directly off the collapse column [Analysis tables]. FastWAM/GR00T reach ~87% correct-first-contact in standard mode but only ~52.8% / 36.0% final success — i.e. picking the right tool is not the binding constraint; getting it to the goal is.
Real-robot fine-tuning on 91 demonstrations partially transfers. GR00T N1.7 and FastWAM reach 4–6/10 on real BallMove / BallRetrieve in both standard and decoy modes after 5,000 updates; ACT reaches ≤2/10 [real-robot table].
ToolBook adds a human-to-robot-retargeting corpus at scale. 3,003 simulation demos (289–388 at L0 per scenario, 100 each at L1/L2) collected via Meta Quest teleop plus 29/21 real-robot BallMove and 20/21 real-robot BallRetrieve demos; released under CC BY-NC 4.0 [ToolBook §].

Method

The benchmark is built on Unitree G1 (humanoid, 36-dim action space) in simulation plus a real G1 for the final-stage evaluation. Each task is parameterized by scenario × execution level × tool-set mode:

Scenarios — Move (push a ball), Retrieve (hook/pull an out-of-reach ball), Break (crack ice). Each scenario needs a different tool function (push rod, hook, hammer-like), forcing functional selection rather than visual template matching.
Execution levels — L0 Select (locate and touch the right tool first), L1 Stationary (use the tool while standing still), L2 Mobile (use the tool while locomoting).
Tool-set modes — Standard (only candidate tools present) and Decoy (candidate tools mixed with same-category confusers that look right but don’t afford the task; e.g. a ‘retrieve’ task gets several rod-shaped distractors).

Policies evaluated: ACT (52.1 M, IL), DP (84.7 M, IL), π0.5 (3.62 B, VLA), Ψ0 (2.63 B, VLA), GR00T N1.7 (3.14 B, VLA, deployed on real), Cosmos 3 Edge Policy (3.37 B, WAM), FastWAM (6.02 B, WAM, deployed on real). All trained with 40,000 updates × 128 samples per update on all 18 tasks jointly; ACT/DP consume language via frozen CLIP ViT-L/14 text encoding; chunk sizes 16–100 depending on head family. Episodes run ≤3,000 control steps at 50 Hz with seeds matched across policies. Three policies (ACT, GR00T N1.7, FastWAM) then fine-tuned on 91 real G1 demonstrations for 5,000 updates and evaluated 10 trials per cell.

ToolBook is collected by eight teleoperators using Meta Quest controllers mapped onto the G1’s upper body + walking primitives; 55 distinct tool meshes span the three functional categories, with per-tool decoys procedurally generated by resampling the same mesh family. Every rollout is scored with five binary event flags along the sub-success chain, which is what makes the selection-vs-execution gap visible.

Results

The headline numbers at the simulation stage (standard mode / decoy mode, success %):

FastWAM — standard L1 Move/Retrieve/Break 76 / 57 / 86; L2 9 / 20 / 69. Decoy L1 75 / 45 / 82; L2 8 / 11 / 61.
GR00T N1.7 — standard L0 77, L1 64 / 36 / 66; L2 10 / 9 / 31. Decoy L0 67, L1 58 / 30 / 63; L2 30 / 5 / 48.
Ψ0 (VLA) — standard L1 9 / 0 / 13; L2 1 / 0 / 2.
π0.5 (VLA) — standard L1 4 / 0 / 12; L2 1 / 0 / 0.
Cosmos 3 Edge Policy (WAM) — standard L0 12, L1 ≤1, L2 0 across the board.
ACT/DP (IL) — ≤5 L0 standard, 0 at L1/L2.

Averaged sub-success chain (standard mode, L1+L2 averaged over the three scenarios): candidate contact 99.8 → correct first contact 87.7 → correct-tool lift 89.2 → tool-target contact 88.2 → success 52.8 for FastWAM; and 99.7 → 87.3 → 90.0 → 79.8 → 36.0 for GR00T N1.7. The sharp drop happens at the “tool-target contact → success” transition: the right tool reaches the target >80% of the time but the task is only completed ~37–53% of the time.

Real-robot (10 trials each, standard / decoy):

GR00T N1.7 BallMove 6 / 4; BallRetrieve 4 / 4.
FastWAM BallMove 5 / 4; BallRetrieve 4 / 5.
ACT BallMove 1 / 0; BallRetrieve 2 / 1.

Probes on a standard-only GR00T N1.7 confirm the two failure modes the design was built to isolate: 85→74% (BallMove) and 59→36% (BallRetrieve) correct-first-contact collapse on unseen tools, and 84% ice-break rate under an unrelated instruction (vs. 77% with the correct instruction).

Why it’s interesting

HumanoidToolBench is the first filed real-humanoid benchmark that jointly grades functional tool selection and whole-body execution with locomotion — sibling but complementary to The Imitator Game: Benchmarking Robot Imitative Ability Beyond Action Prediction which level-grades the imitation gap, In-context Robot Learning Made Simple: A Democratized Recipe for Manipulation Tasks which level-grades affordance disambiguation, and Reality Check — Pokeandwiggle's Real-World Robot Manipulation Leaderboard on an FR3 Duo Farm which standardizes the real-hardware eval surface. The sub-success chain decomposition is the most actionable new diagnostic the paper contributes: it makes the selection ↔ execution gap directly readable rather than hidden inside a single task-success number.

The specific head-to-head is the sharpest filed WAM-vs-VLA comparison on a tool-use benchmark: FastWAM beats GR00T N1.7 at execution despite GR00T leading at pure selection, mirroring the Dyna-2: A 1-Million-Hour Scaling Law for World-Action Models Dyna-2 and Fast-WAM: Do World Action Models Need Test-time Future Imagination? FastWAM positions that WAM backbones generalize to deployment regimes VLAs under-perform on. The instruction-ignoring probe (84% breaks ice under “the weather is nice today”) is empirical reinforcement of the Breaking the Vision-Action Shortcut: Latent Interface Training for Generalizable Robotics Foundation Models / Anchor-Align: Generalizable VLA Finetuning via Representation Anchoring and Language-Action Alignment shortcut-learning thesis — the policy is reading scene geometry, not language. ToolBook adds a dexterous-humanoid teleop corpus with explicit sub-success labels to the Synthetic Training Data substrate pool, usable as training data for Human-to-Robot Retargeting recipes that need level-graded supervision.