HumanCLAW: Can Vision-Language Models Act Through a Body?

ListenThe Body It Now Controls10:11

Show notes

HumanCLAW is an evaluation framework that isolates a vision-language model's *action intelligence* — its moment-to-moment choice of what a body should do next — from the low-level motor-control problem. Across nine frontier VLMs the best reaches just 16.8% interaction success, and the dominant failure isn't perception but *embodied self-awareness*: models lose track of where their own body is. We unpack the decouple-decision-from-execution design, the stage-by-stage collapse, and the ablations showing that structured memory and a small verifier beat simply feeding more context. **Sources** - HumanCLAW: Can Vision-Language Models Act Through a Body?

Subscribe to the channel: https://podcast.lumalabs.link/27a25f51ea81d41a8253f46c6133a35e/feed.xml

TL;DR

HumanCLAW is an evaluation framework that isolates a vision-language model’s action intelligence — its moment-to-moment decision of what a physical body should do next — from the low-level motor-control problem. At every 0.5 s step, a frozen off-the-shelf VLM sees an egocentric RGB view plus a registered skill list, and emits one atomic skill command; a motion-continuation DiT + per-skill ControlNet adapters translate the command into ~2 s of full-body motion executed kinematically against a physically responsive world (gravity, collisions, movable objects). The companion HumanCLAW-Bench comprises 1,218 long-horizon find-navigate-interact episodes across 41 indoor HSSD houses over six target categories, scored by progressive Find/Nav/Interact success rates plus body-awareness and motion-quality metrics. Across nine frontier VLMs the best (Gemini-3.1) reaches only 16.8% InteractSR and four models score at most 0.2%; the dominant failure mode is not object recognition but embodied self-awareness — the VLM loses track of where its own body is, whether it has arrived, and whether it has hit an obstacle.

Key claims

Decouple decision from execution: at every 0.5 s the VLM issues an atomic skill command, and a motion-continuation DiT with per-skill ControlNet adapters realizes it kinematically inside a physically responsive world, so failures can be attributed to the model’s choice rather than to balance/tracking bugs in a locomotion controller [§Method, project page].
Frontier VLMs collapse on the full task: Gemini-3.1 tops the leaderboard at 64.9% FindSR, 42.5% NavSR, 16.8% InteractSR; four of the nine tested VLMs (Qwen3.6-35B-A3B, Qwen3.5-27B, InternVL3.5-38B, and Qwen3.6-27B at 0.2%) score ≤0.2% InteractSR [Table 1].
The dominant deficit is embodied self-body-awareness, not perception: once the target actually enters the ego view it is almost always recognized; navigation-stage failures split into 34% attributable to body-awareness errors and interaction-stage failures 81% attributable to body-awareness errors, surfacing as termination decisions that misfire in both directions (stop too early / never confirm arrival) [project page].
Recognition-vs-arrival split: of 5,473 episodes where the agent actively finds the target, 68% still fail to reach it — nearly two-thirds of those from egocentric self-spatial-awareness errors in the stopping decision rather than from failing to see the goal [project page].
Contact-to-success drop-off ranges from 90% (Gemini-3.1) down to 3.5% (InternVL3.5-38B) given navigation succeeded, so an order of magnitude of variance sits in issuing and confirming the terminal sit skill at the right moment, not in executing it — the motion layer realizes the sit reliably when it is issued [project page].
Body-part collision profile: legs and feet collide in 28–45% of steps, arms and hands in 20–35%, the head almost never (<7%) — collisions concentrate exactly where the ego view does not directly see the body, evidence that the VLM behaves “fluent about the world but oblivious to the body it now controls” [project page].
Structural memory beats raw context length: adding a compact structured memory lifts NavSR from 11% (no history) to 27%; a mid-level 2–3 s objective binding multi-step routines lifts InteractSR from 0% to 18.9%; a short-context spatial-action outcome verifier lifts NavSR from 2% to 27%. Merely lengthening text or visual history saturates and ten image frames actively hurt [project page ablations].
Body dataset & benchmark scale: 1,218 episodes over 41 HSSD houses × six target categories (chair, bed, couch, potted plant, toilet, TV), stratified by three geometric difficulty dimensions (distance, choice, obstacle); movable objects are marked dynamic so contact displacement is measurable [project page].

Method

The evaluation loop is: at every 0.5 s the frozen VLM receives system prompt + task instruction + registered skill list & rules + text history + current ego view, then reasons in three stages — percept & understand (spatial layout / target visibility / obstacles into a visual state), dynamic mid-level plan (a 2–3 s objective inherited or revised from the last step), and low-level skill proposal (one atomic skill plus parameters). A short-context spatial-action outcome verifier interrogates the proposal with skill-specific questions and accepts, rejects, or renews it from the skill pool before it reaches the body.

Execution: a motion-continuation DiT takes 5 history poses and denoises the next 15 future frames as a velocity field, producing continuous full-body motion chunk-by-chunk. Each atomic skill has its own lightweight ctrl-DiT adapter fed by a skill-condition encoder, arranged as a plug-and-play ControlNet bag — the VLM’s skill<param> decision routes dynamically to the matching adapter, and new skills can be registered without retraining the motion prior. Generated motion is executed kinematically while the world responds physically: walls block the body, touched objects displace, gravity + ground contact shape stair-climbing. Balance and motor-tracking failures are factored out by construction.

The benchmark is a progressive find→navigate→interact task (“Look for <obj>, navigate to it with zero distance, and finally sit on it”) over 41 HSSD validation houses × six target categories × three geometric difficulty dimensions (distance, choice, obstacle), yielding 1,218 episodes with dynamic-tagged movable furniture so disturbance is measurable.

Results

Headline leaderboard (InteractSR / NavSR / FindSR): Gemini-3.1 16.8 / 42.5 / 64.9; Gemma-4-31B 11.1 / 28.7 / 58.1; Gemini-2.5 3.5 / 21.6 / 58.5; GPT-5.5 3.4 / 14.1 / 55.8; Claude-4.8 1.5 / 8.7 / 32.7; Qwen3.6-27B 0.2 / 20.9 / 51.0; Qwen3.6-35B-A3B 0.0 / 5.8 / 44.6; Qwen3.5-27B 0.0 / 13.5 / 37.8; InternVL3.5-38B 0.0 / 0.8 / 46.8 [Table 1]. Every model shows the same sharp stage-by-stage drop — most of the task is lost between finding and reaching, and again between reaching and interacting. Collision-step fractions cluster at 34–51% across models. Motion jerk (root-rigid jerk at ~0.27 s timescale, lower = smoother) is lowest for GPT-5.5 at 4.2 and highest for Gemini-2.5 at 8.7 — decoupled from success rate, evidence that motion smoothness is not the bottleneck. Token cost per step ranges from 4,360 in / 354 out (GPT-5.5) to 7,047 in / 625 out (Claude-4.8).

Ablation deltas (project page): structured memory NavSR 11% → 27%; mid-level objective InteractSR 0% → 18.9%; short-context verifier NavSR 2% → 27%. Ten image frames hurt; pure-text CoT saturates. 38% of Find failures are ineffective exploration vs 23% recognition failures. 34% of Nav failures and 81% of Interact failures are attributable to body-awareness errors.

Why it’s interesting

HumanCLAW is the first filed benchmark that isolates the VLM’s body-controlling half from the motor-control half, and it lands hard in the failure-mode literature: VLM Perception Failures has documented that VLMs answer vision-centric questions from LLM priors (Hidden in plain sight: VLMs overlook their visual representations) and collapse on pixel-precise 3D-grounded output (BOP-Ask: Object-Interaction Reasoning for Vision-Language Models) — HumanCLAW adds a fourth mode, embodied-self-body-awareness failure, and provides the clearest numeric evidence yet that the deficit is neither perception nor motor control. The benchmark is also a rare structural counterpart to VLA Models: rather than train a VLA end-to-end, it wraps a frozen off-the-shelf VLM as the decision half and a small motion-prior + ControlNet-bag as the execution half — the same “wrap the frozen VLM as an orchestrator” pattern Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents used for manipulation, pushed here to whole-body humanoid navigation with a much simpler skill library. Anthropic’s Go2 field report (How Claude Performs on Robotics Tasks) documented the same kind of failure anecdotally (Claude walks into a trash can because it is “to the left of the crosshair”); HumanCLAW gives it a benchmark.