Addressing the Orchestration Gap in Generalist Robots via Physical Agency (Pigey)

TL;DR

Pigey (Physical Agency) is a closed-loop physical-agent orchestrator that decomposes generalist robot behavior into a high-level agent manager plus a stable of frozen low-level control agents — a VLA policy (π0.5) for contact-rich manipulation and a TAMP backend for rigid pick-and-place — without any additional data collection, policy post-training, or weight updates. The orchestrator plans subgoals, routes each one to the appropriate frozen skill, verifies the outcome from gripper sensors and the wrist camera, and recovers when a step fails. On 30 real-robot Franka FR3 tasks Pigey lifts π0.5 from 16.7% → 97.3% and beats the strongest orchestrator baseline (TiPToP, 48.7%); on LIBERO-PRO the same frozen π0.5 goes from 12.8% → 53.3%, over 4× the base policy and a new state of the art. The paper names the difference between what frozen motor skills achieve alone and inside the agentic loop the orchestration gap.

Key claims

Generalist-robot capabilities — perception, world knowledge, planning, success detection, recovery, low-level control — can be decomposed into a general language-conditioned policy/control agent and a high-level agent manager/orchestrator, rather than folded into a single end-to-end learned policy via large-scale pretraining [Abstract].
Pigey’s orchestrator runs an explicit observe → reason → act → verify → recover loop, with the frontier VLM planning subgoals, routing each to a frozen motor skill, checking the outcome from the gripper sensor and wrist camera, and recovering on failure — the weights of the underlying policies never change [Tweet thread 1/N–3/N].
Two frozen backends are held simultaneously and the orchestrator picks between them per subgoal: TAMP for rigid pick-and-place and grasp planning, and π0.5 (a flow-matching VLA) for deformables and contact-rich motion; the orchestrator sequences skills but never emits low-level motor commands itself [Tweet thread 4/N].
On 30 real-robot Franka FR3 tasks, Pigey reports 97.3% vs TiPToP 48.7% vs π0.5-direct 16.7%, with the gains landing on the reasoning-limited task categories (safety, obstacles, world knowledge, conditional logic, multi-step, spatial, recovery, long-horizon memory) [Tweet thread 5/N].
On LIBERO-PRO with the same frozen π0.5 weights and no task-specific finetuning, Pigey reports 53.3% vs CaP-Agent0 18.2% vs π0.5-direct 12.8% — over 4× the frozen policy and a new SOTA [Tweet thread 6/N].
The paper defines the orchestration gap as the difference between what frozen motor skills achieve alone and what they achieve inside the agentic loop, arguing that closing it is a distinct axis from the standard scale-the-VLA lever [Abstract].

Method

Pigey is a closed-loop orchestrator built around a frontier VLM as the high-level agent manager and a fixed roster of frozen low-level agents: a TAMP planner for rigid pick-and-place and a π0.5 VLA for contact-rich or deformable manipulation. The orchestrator decomposes a natural-language task into a sequence of subgoals, picks which frozen backend to invoke for each subgoal, issues that subgoal to the backend, and then reads back both the gripper sensor and the wrist camera to verify whether the subgoal succeeded. On failure, the orchestrator diagnoses the outcome and produces a recovery plan (retry with a modified subgoal, switch backends, or replan globally); the demonstrations described in the announcement thread include memorizing a table of five near-identical dolls, looking away while an operator shuffles them, and putting each doll back — reordering the placements online when a destination is still occupied. Because no low-level agent is finetuned, the entire system depends on the VLM’s ability to characterize where each frozen primitive is reliable and when to intervene — the load-bearing move is routing plus verification plus recovery, not skill acquisition.

Results

30 real-robot Franka FR3 tasks: Pigey 97.3%, TiPToP 48.7%, π0.5 direct 16.7% — an 80.6-pp lift over the base VLA on tasks explicitly designed to be reasoning-limited (safety, obstacles, world knowledge, conditional logic, multi-step, spatial, recovery, long-horizon memory) [Tweet thread 5/N].
LIBERO-PRO (same frozen π0.5 weights, no task-specific finetuning): Pigey 53.3%, CaP-Agent0 18.2%, π0.5-direct 12.8% — a 4.16× lift over the base policy and a new state of the art on this simulation benchmark [Tweet thread 6/N].
Long-horizon memory demo: memorize five near-identical dolls, tolerate an off-view shuffle, and restore every one — including online re-ordering of the plan when the destination is blocked [Tweet thread 7/N].
Ablation-level detail (which of routing / verification / recovery contributes most, how sensitive the gains are to the choice of frontier VLM, whether TAMP alone or π0.5 alone under the same orchestrator would already close much of the gap) is not visible from the abstract or thread; the paper body will need a targeted refresh once the PDF is parseable.

Why it’s interesting

Pigey is the sharpest quantitative case yet for the “wrap the frozen VLA in an orchestrator” recipe that VoLo: A Physical Orchestrator for Open-Vocabulary Long-Horizon Manipulation introduced as Physical Orchestration and that Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents pushed to a contact-only VLA invocation on the same LIBERO-PRO benchmark. Where Harness VLA reports +38.6 pp on LIBERO-PRO with a frozen π0.5, Pigey reports +40.5 pp on the same benchmark with the same base policy (12.8 → 53.3), and adds a real-robot number (16.7 → 97.3 on 30 Franka tasks) that neither VoLo nor Harness VLA staked out — evidence the orchestration lever compounds hard on reasoning-limited real hardware where the frozen policy is near-zero. The move that most sharply distinguishes Pigey from the existing wiki cluster is multiple heterogeneous frozen backends inside one orchestrator (VLA + TAMP), sibling to RoboHarness: Memory-Driven Orchestration of Heterogeneous Robot Policies for Long-Horizon Planning‘s Memory-Bridge routing across VLAs / RL / TAMP but with the paper’s own contribution focused on the closed-loop verify-and-recover half rather than the trajectory-conditioned routing half. It also lands cleanly in the Tool-Use Agents design space — the frontier VLM is running a ReAct-like loop where the “tools” are typed physical primitives — and adds a real-hardware datapoint to the pattern CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation documented (OpenVLA and π0 collapse on LIBERO-PRO under position and instruction perturbations) as motivation. Naming the gap between what frozen skills achieve alone versus in the loop the orchestration gap sharpens the recipe debate: the concept-page bet has been on which base VLA to train and how, and Pigey argues a separate, measurable, sometimes-larger axis is what to wrap the trained VLA in at deployment.