GaP: A Graph-as-Policy Multi-Agent Self-Learning Harness For Variational Automation Tasks

TL;DR

GaP (Berkeley/NVIDIA/CMU; Goldberg, Fan, Zhu, Sastry) introduces Variational Automation (VA) — persistently repeated tasks with bounded but non-trivial variation in object geometry and pose — as a new problem regime between fixed automation and generalist robotics, and answers it with a multi-agent coding harness whose output is a directed computation graph of typed skill nodes rather than either a monolithic VLA policy or a linear code trajectory. An orchestrator agent partitions a natural-language task into segments, skill agents synthesize localized subgraphs from a 51-skill Modular Open Robot Skill Library (MORSL), and a parallel-simulation rehearsal loop autonomously edits topology + parameters using contact and state feedback until success plateaus. Across 8 new VA benchmarks (4 sim, 4 real — grocery fulfillment, grocery packing, popcorn, USB-C insertion, bimanual crate washing), GaP holds 0.93–0.99 where π₀.₅ / MolmoAct2 / CaP-X drop to 0.05–0.24 under pose variation, matches an expert hand-engineered bimanual graph (0.953 vs 0.987 on crate washing), and stages VLA policies into distribution for >2× success — an interpretable middle path between TAMP and end-to-end VLAs.

Key claims

Variational Automation is defined as a distinct problem regime — a robot persistently executing varying instances of a task within a known, bounded workcell where object range and poses are non-trivial but not open-world — sitting explicitly between fixed automation (identical repetition) and full generalist robotics (open-ended VLAs) [§What is Variational Automation].
A policy in GaP is a directed computation graph of atomic perception / grasp-planning / motion-planning / vision / ROS-translation / verification nodes joined by statically type-checked data and control edges — interpretable, modular, and reusable, inspired by ROS computation graphs and TAMP hierarchies [§Graph-as-Policy].
MORSL exposes 51 initial skills, each declaring graph inputs, outputs, parameters, and pre-conditions in agent-readable conventions — the coding agents compose these into subgraphs rather than emitting free-form Python [§MORSL skill library].
Separating orchestration from per-skill authoring — one orchestrator agent partitions the task and dispatches skill agents that synthesize localized subgraphs, with every edge statically type-checked — limits per-agent context and reduces incentives to “cheat” verification [§Multi-agent harness].
Self-learning rehearsal runs parallel graphs in a parameterized simulator, registers pre/post node states, and uses contact + state feedback to localize failures to specific nodes so agents edit topology and parameters until performance plateaus [§Self-learning rehearsal].
Positional robustness. On sim VA benchmarks with larger pose variation (X-Y 20×20, basket_swap, permutation, mixed_all, Pack fixed / varied), GaP scores 0.93–0.99; π₀.₅ scores 0.15–0.78, MolmoAct2 0.10–0.90, CaP-X 0.01–0.11 — the VLA baselines collapse where LIBERO-standard pose variance is exceeded [§Results, sim table].
Staging VLAs. Prefixing GaP as a wrist-camera stager before a VLA more than doubles VLA success on the harder VA columns: π₀.₅ 0.15 → 0.32 (basket_swap), 0.17 → 0.67 (Pack fixed); MolmoAct2 0.26 → 0.58 (basket_swap), 0.20 → 0.66 (mixed_all) [§Results, sim table rows 5–6].
Real robot. On real Franka VA tasks GaP reaches 25/25 grocery fulfillment, 28/30 grocery packing, and 18/20 make-popcorn vs a TipTop baseline at 8/25, 10/30, 0/20; average single pick-and-place execution time is 67 s (GaP) vs 95 s (TipTop) [§Results, real-robot table].
Self-learning works. Popcorn success rises 33% → 94% in sim and 90% (18/20) real over 10 rehearsal iterations of autonomous graph editing driven by contact-feedback failure localization [§Rehearse, localize failures, refine the graph].
Approaches expert engineering. On a bimanual industrial crate-washing cell (150 sim trials + 3-hour continuous throughput run), GaP scores 0.953 vs hand-engineered expert 0.987 at nearly matched cycle time (179.13 s vs 176.47 s) — 18.33 vs 19.33 successes/hour [§Results, crate-wash table].
Ablation 1 — graph beats graphless. Replacing the graph with a single LLM emitting raw Python collapses success to zero on all VA tasks; interface and syntax mismatches terminate runs even when high-level logic is right [§Results ablation].
Ablation 2 — multi-agent decomposition matters. Condensing the specialized authoring agents into one LLM also drops success to zero; every trial fails static structural verification [§Results ablation].
Cycle times remain above the ~7 s/instance industrial standard; the paper flags fewer VLM calls, faster IK/motion planning, and more self-learning iterations as required next steps [§Industrial reliability & throughput].
The benchmarks focus on quasi-static pick-and-place; only USB-C insertion uses force feedback. Deformables, dynamics, and moving targets are flagged as future work [§Beyond quasi-static].

Method

GaP is a four-layer stack. (1) MORSL — a Modular Open Robot Skill Library of 51 atomic nodes covering perception (SAM3 / Molmo2 pointing / open-vocabulary detection), grasp planning (AnyGrasp-style proposals + filters), motion planning + IK (with reachability / collision checks), 2D/3D vision utilities, ROS translation, and verification. Each node declares typed inputs, outputs, parameters, and pre-conditions in an agent-readable schema.

(2) Multi-agent harness — an orchestration agent reads the natural-language task and geometric object models, partitions the task into skill-aware semantic segments, and dispatches skill agents to synthesize localized subgraphs from MORSL. The orchestrator then wires the subgraphs into one executable graph. All edges are statically type-checked; a graph that fails structural verification never reaches the simulator.

(3) Internal simulation rehearsal — the harness spins up a parameterized simulator instance per candidate graph, executes parallel rollouts with pre/post state and contact-feedback logging at every node boundary, and localizes failures to specific nodes. A rehearsal agent proposes topology edits (add/remove/replace nodes, rewire control flow) and parameter edits, then relaunches parallel rehearsals. The loop runs until success on held-out instances plateaus.

(4) Edge interpreter — the frozen optimized graph G* is shipped to a lightweight interpreter running on the physical robot; interpretability + reusability come for free from the graph structure. Baselines evaluated in the same harness: CaP-X (linear Code-as-Policy from CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation), TipTop, π₀.₅, MolmoAct2.

Results

Sim VA benchmarks (5,500 trials, 100 instances per cell):

Method	LIBERO	LIBERO-Pro	X-Y 20×20	basket_swap	permutation	mixed_all	Pack fixed	Pack varied
CaP-X	—	0.22	0.07	0.05	0.11	0.10	0.01	0.01
π₀.₅	0.96	0.24	0.78	0.15	0.20	0.20	0.17	0.18
MolmoAct2	0.97	0.43	0.90	0.26	0.10	0.20	0.18	0.18
π₀.₅ + GaP (stager)	0.85	0.60	0.79	0.32	0.50	0.39	0.67	0.66
MolmoAct2 + GaP (stager)	0.70	0.62	0.84	0.58	0.39	0.66	0.59	0.59
GaP	0.95	0.95	0.95	0.97	0.93	0.97	0.99	0.98

Real-robot Franka tasks: GaP reaches 25/25 / 28/30 / 18/20 on Fulfill Grocery Orders / Pack Grocery Items / Make Popcorn vs TipTop 8/25 / 10/30 / 0/20 [§Results, real table]. USB-C cable insertion on UR5 with force feedback: 32/35 ascending, 32/35 descending, 20/20 odd, 14/15 even ports per trial [§Results, USB table]. Bimanual crate washing (150 trials + 3h continuous run): 0.953 success / 179.13 s cycle / 18.33 succ/hr vs hand-engineered expert 0.987 / 176.47 s / 19.33 succ/hr [§Results, crate table]. Self-learning on Make Popcorn: 33% → 94% sim / 90% real over 10 iterations of autonomous graph editing [§Rehearse…].

Why it’s interesting

GaP is the same Berkeley / Goldberg lab’s third filed swing at “how to compose a robot policy from a coding-agent harness plus a skill library” — after CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation (CaP-X = linear Code-as-Policy + auto-synthesized skill library) and Playful Agentic Robot Learning (RATs = pre-task play to grow the skill library) — and it makes the sharpest structural move: retire the linear program and replace it with a directed graph that a multi-agent orchestrator composes and a self-learning loop refines. That the ablations collapse to zero when either the graph is removed or the multi-agent decomposition is collapsed is unusually strong internal evidence that both structural choices are load-bearing, not incidental. The staging result (>2× VLA success by prefixing GaP as a wrist-camera positioner) reframes GaP as complementary to VLA Models rather than a pure counter-recipe: coarse-scale planning + fine-scale action can be different subsystems. It also completes an interesting triangle in the AI-for-AI Research cluster — ENPIRE: Agentic Robot Policy Self-Improvement in the Real World (coding agents self-improve a fixed VLA via real-robot RL), RATs (coding agents self-acquire a task-agnostic skill library via play), GaP (coding agents self-compose and self-refine a task-specific graph via rehearsal) — three closed-loop harnesses that pay their action costs on physical or simulated robots rather than on GPU clusters. Relative to Tool-Use Agents (SkillSynth’s Toward Scalable Terminal Task Synthesis via Skill Graphs, InfTool’s Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing), GaP moves the “structured trajectory generator” idea into the output space itself: the graph is not a way of sampling training data, it is the policy.