Show-Harness: Just a VLM Agent Can Play Robots

TL;DR

Show-Harness (Show Lab, NUS) is a semantic-action-interface Embodied Harness that turns any frontier VLM into a robot controller by exposing a small vocabulary of discrete action units (e.g. MV_LEFT, GRASP) whose metric magnitudes are supplied by embodiment-specific interpreters — not the VLM. The paper argues (and ablates) that the interface itself is the action space: unit names carry model-known conventions, and stripping both name and convention drops zero-shot success from ~95% to 5%. The same interface doubles as GUMI, a GUI-based demonstration harness that lets humans or agents “play” robots without teleoperation hardware, and both closed-source zero-shot VLMs (Gemini-3.1 Pro) and small fine-tuned open models (Qwen3.5-2B) beat all three baseline families — VLA (π0.5, GR00T), VLA-centric agents (H-VLA, G-VLA), and code-as-policy (CaP-X, RATS).

Key claims

The action-unit vocabulary is load-bearing, not decoration. Named units + spelled-out conventions reach ~95%; arbitrary symbols with conventions reach 100%; names alone (no convention) still reach 90%; strip both and success collapses to 5% with 19/20 timeouts, and the agent gets only 23.3% of six-direction mappings right against a 16.7% chance rate [§Project page — interface ablation].
A zero-shot frontier VLM (Gemini-3.1 Pro) matches or beats every baseline family on every environment shift — 20/20 across held-out Teddy and Chess tasks — while a fine-tuned Qwen3.5-2B, trained only in simulation, still transfers to real at 13/20, versus 0/20 for both trainable VLA baselines (π0.5, GR00T) under the same real-transfer protocol [§Project page — main results].
Adaptation is a change to the interpreter, not the model. A 1 cm interpreter step lifts zero-shot 60→80% and fine-tuned 40→65% with no retraining, where π0.5 needs its full budget on fine-grained data to reach 60%; new rotation units extrapolate to unseen 90° at 10/10 for ZS vs 20% for π0.5; joint dual-arm prediction lifts avg 50→80% and removes every collision [§Project page — adaptation].
The zero-shot ceiling is limited by fine-grained spatial grounding, not planning or language understanding. Five frontier VLMs under the same harness split into two tiers (86–96% vs 72–78%); instruction following (≥98%) and subtask plans (≥97%) are near-saturated across the board, while empty grasps climb from 6% to 26% down the ranking — the models miss on the centimetres [§Project page — frontier VLM analysis].
Above 2B parameters, capacity buys precision but costs latency; on moving targets latency wins. Success is flat 88–92% across Qwen3.5-2B/4B/9B and InternVL3.5-2B/8B; precision-bottlenecked tasks (stacking, peg insertion) scale 50→65→75% from 0.8B to 9B; but on a rolling tennis ball 2B (80%) beats 9B (60%) because 9B’s 79 ms/decision cannot track motion vs 2B’s 39 ms [§Project page — backbone scaling].
GUMI closes the collection loop with the same interface — every action unit is a labeled key, so a human at a keyboard and a GUI agent reading the same screen drive the robot identically, and every step is a ready training pair with no specialized teleoperation hardware [§Abstract, §Project page — GUMI section].
Plugin decomposition shows which components carry the load. Removing one default plugin at a time from the 96% configuration: Multi-View Guidance is worth 38 points, Subtask Planning 36, Proprioception 28, Failure Recovery 24, Action History 20 (with 23% fewer steps); Action Chunking holds 96% at 19% fewer VLM calls; Adaptive Step hits 96% in 30 steps vs 82% for fixed 2 cm and 70% for fixed 4 cm [§Project page — plugin ablation].

Method

Show-Harness reformulates robot control into parameter-free discrete semantic action units — each a bare symbol whose metric magnitude is supplied by an embodiment-specific interpreter. The same vocabulary drives a 7-DoF Franka, a 6-DoF AgileX, and a simulator without changing the model. A unit like MV_LEFT carries a name the VLM already knows plus a convention fixing whose left it is; ablations confirm both the name-based prior and the spelled-out convention are load-bearing, and arbitrary symbols work only if the convention is provided. Around this interface, the harness ships a set of plugins (Multi-View Guidance, Subtask Planning, Proprioception, Failure Recovery, Action History, Action Chunking, Adaptive Step, Visual Prompt, Situated Planning) that trade cost for coverage — the first five are on by default and each is worth 20–38 points; the rest activate for specific regimes. Two model routes share the harness: a zero-shot agent using Gemini-3.1 Pro (or GPT-5.6-sol, Opus 5, GPT-5.6-luna, Gemini-3.6-flash), and a fine-tuned small VLM (Qwen3.5-2B — a few GPU-hours of SFT on GUMI-collected data). GUMI (GUI Manipulation Interface) mirrors the action space onto keyboard keys so demonstrators — human or agent — drive the robot through the same interface used at deployment, and every recorded keystroke is a labeled training pair.

Results

Zero-shot Gemini-3.1 Pro reaches 20/20 across every environment shift including held-out Teddy and Chess tasks, beating all three baseline families (VLA: π0.5, GR00T; VLA-centric agents: H-VLA, G-VLA; code-as-policy: CaP-X, RATS) [§Project page].
Fine-tuned Qwen3.5-2B, trained only in simulation, transfers to a real robot at 13/20; π0.5 and GR00T both hit 0/20 under the same protocol [§Project page].
Frontier VLM tier split: Gemini-3.1 Pro / GPT-5.6-sol / Opus 5 at 86–96%; GPT-5.6-luna / Gemini-3.6-flash at 72–78%. Empty-grasp rate climbs 6% → 26% down the ranking; instruction following ≥98% and subtask plan ≥97% across the board.
Thinking effort mostly shortens episodes (GPT-5.6-sol: 37 → 30 steps) at 3.4× wall-clock cost; planning is not the gap.
Interpreter-change lifts: 1 cm step 60→80% ZS and 40→65% FT with no retraining; unseen 90° rotation 10/10 ZS vs 20% for π0.5; joint dual-arm prediction 50→80% avg and eliminates collisions.
Backbone scaling: Qwen3.5 at 2B/4B/9B and InternVL3.5 at 2B/8B all 88–92% (within 4 pp); 0.8B and 1.5B hesitate (72–74%, episodes 37.6→57.8 steps). On rolling tennis ball 2B (80%) > 0.8B (70%) > 9B (60%) because 9B’s 79 ms decision latency cannot track motion vs 2B’s 39 ms.
Plugin ablation from 96% baseline: −Multi-View 38 pp, −Subtask Planning 36 pp, −Proprioception 28 pp, −Failure Recovery 24 pp, −Action History 20 pp (with 23% fewer steps). Action Chunking preserves 96% at 19% fewer VLM calls; Adaptive Step hits 96% in 30 steps vs 82% for fixed 2 cm / 70% for fixed 4 cm; Visual Prompt and Situated Planning lift their target case 35→85%.

Why it’s interesting

Show-Harness is the sharpest published instance yet of the LLM/VLM-as-controller row of the VLA recipe-lever board — sibling in framing to the Robocurve / Astra / Fable evidence thread (Jay Chooi (Robocurve) — Astra vs Fable 5.1 on real-robot manipulation via IK, Comparing Fable 5.1 to Fable 5 on robotic manipulation, thijs — Astra paints the Golden Gate Bridge with a real robot arm, Wenli Xiao — GPT-6 Astra does physical in-context learning from a single human video via a motion-planning tool harness) and to Robot-Use Agents‘s “robot-use agents” positional post, but with quantified ablations that isolate what about the interface makes it work. The interface-is-the-action-space ablation (95% → 5% when both name and convention are stripped) is a direct empirical answer to a question left open across VLA Models and Tool-Use Agents — how much of LLM-as-controller success is model capability vs interface engineering — and lands closer to “mostly interface”. The GUMI half is the collection-side mirror: puts Show-Harness alongside Human-to-Robot Retargeting entries that make the collection tool embodiment-invariant (Solving Dexterity: A Full-Stack Approach (mimic hand M1 + wearable U1), HandUMI — Robot-free bimanual data collection retargetable to any parallel-gripper arm (Brik H. Meza launch tweet), Xiaomi-Robotics-1 (XR-1) — Scaling VLA Foundation Models with 100K Hours of Embodiment-Free UMI Pre-training), but with the invariance living in a GUI action-key vocabulary rather than a physical device. Finally, the “fine-grained spatial grounding is the ceiling, not planning” diagnosis quantifies a claim the VLM Perception Failures cluster has been building toward and points at where the next capability jump on this row has to come from.