CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation

TL;DR

CaP-X is a robot-manipulation framework with three components: CaP-Gym (a Gymnasium-style environment that wraps Robosuite / LIBERO-PRO / BEHAVIOR behind a Python REPL exposing perception primitives like SAM3 and Molmo2 and control primitives like IK / motion planners), CaP-Bench (8 evaluation tiers — S1-S4 single-turn × abstraction levels, M1-M4 multi-turn × feedback modalities — over 7 core tasks, run against 12 frontier LMs/VLMs), and two improvement axes derived from the benchmark — CaP-Agent0 (training-free agentic scaffolding) and CaP-RL (online GRPO on the coding agent itself). The headline empirical finding is that test-time compute scaling can close the gap that low-level primitives open: agents on raw low-level APIs with multi-turn feedback and a Visual Differencing Module (M4) reach parity with high-level multi-turn (M3), and CaP-Agent0 matches or exceeds human-expert programs on 4 of 7 tasks while remaining training-free. CaP-RL then post-trains Qwen-2.5-Coder-7B on environment rewards (20% → 72% in simulation, transferring to a Franka with 84% cube-lift / 76% cube-stack zero-shot).

Key claims

Across 12 frontier LMs/VLMs in single-turn Pass@1 (S4, low-level primitives, no usage examples), closed-source models consistently outperform open-source models and newer models are stronger, but none match human-expert reference programs — a persistent gap [§3.3 Takeaway 1, Fig. 1].
Task success rises monotonically with primitive abstraction level (S4 → S3 → S2 → S1), most steeply for weaker / open-source models whose compilation rates collapse at low abstraction [§3.3 Takeaway 2, Fig. 3].
Multi-turn execution feedback (stdout/stderr only, M1) consistently improves performance across all models — explicit execution traces are a load-bearing feedback signal for debugging and recovery [§3.3 Takeaway 3, Fig. 5].
Counter-intuitively, naive multimodal feedback (raw RGB interleaved each turn, M2) degrades performance vs text-only M1; attributed to a cross-modal alignment gap between code synthesis and physical-task images [§3.3 Takeaway 3].
The Visual Differencing Module (M3) — a VLM converts each observation into structured text describing the scene and the delta from the previous frame — substantially outperforms both naive image interleaving (M2) and execution-only feedback (M1) [§3.3 Takeaway 3, Fig. 5].
M4 (low-level primitives + multi-turn + VDM) reaches parity with M3 (high-level primitives + multi-turn + VDM), supporting an explicit test-time compute scaling hypothesis: at runtime, multi-turn reasoning + verification recovers what low-level primitives lose [§3.3 Takeaway 3, Fig. 6].
CaP-Agent0 combines three components — multi-turn Visual Differencing, an auto-synthesized persistent skill library built by extracting recurring functions from successful S3 rollouts, and parallel reasoning (9 candidate generations per turn, single-model or multi-model GPT-5.2 / Claude Opus 4.5 / Gemini-3-Pro, with a central coding agent synthesizing the final code) — and matches or exceeds human-expert programs on 4 of 7 CaP-Bench tasks [§4, Fig. 8].
On LIBERO-PRO with position and instruction perturbations, OpenVLA and π0 score 0% across libero-object / libero-goal / libero-spatial; π0.5 reaches 0.17–0.38 on Pos but ≤0.01 on Task perturbations; training-free CaP-Agent0 reaches 0.22 / 0.26 / 0.12 (Pos) and 0.18 / 0.17 / 0.14 (Task), demonstrating robustness to instruction variation that learned VLAs lack [Table 2].
On long-horizon BEHAVIOR mobile manipulation (R1Pro humanoid, Pick up Radio / Pick up Soda Can, 25 trials each), CaP-Agent0 closes most of the gap to human experts on task success: 56% vs 36% (human) on Radio and 72% / 72% on Soda Can, vs S3 baselines at 24% / 32% [Table 3].
CaP-RL applies GRPO with verifiable environment rewards to a Qwen-2.5-Coder-7B coding agent inside CaP-Gym, lifting average sim success 20% → 72% in 50 iterations and transferring to a real Franka at 84% on cube lifting and 76% on cube stacking — approaching human-expert performance with minimal sim-to-real gap [§5, project page].
The recommended evaluation discipline that follows from Takeaways 1-2 is: evaluate generalist embodied coding agents primarily at low-level primitive performance, since high-level human-designed primitives mask failures in low-level reasoning and impose a generality ceiling [§3.3 Takeaway 2 closing paragraph].

Method

CaP-Gym binds a low-level environment loop (a physics simulator or the real robot) with a stateful Code Executor loop, exposing both as a REPL. Each “turn” is the full execution of an agent-generated Python program, which may invoke many underlying sim steps. All compute-heavy perception and control primitives are stateless services for high-throughput parallel evaluation: SAM3 for language-conditioned segmentation, Molmo 2 for open-vocabulary pointing, OpenCV / Open3D as standard vision libraries, PyRoki for motion planning / IK with collision checking and reachability. CaP-Gym integrates 187 tasks from Robosuite, LIBERO-PRO, and BEHAVIOR.

CaP-Bench evaluates along three orthogonal axes — abstraction level (high-level human-crafted macros vs atomic low-level APIs), temporal interaction (single-turn vs multi-turn with execution feedback), and perceptual grounding (text-only stdout/stderr, raw RGB, or the Visual Differencing Module). The 8 tiers (S1-S4 single-turn, M1-M4 multi-turn) instantiate this design space; primary analysis is over 7 core tasks with 100 trials per tier per task. Models include closed (Gemini-3-Pro, GPT o1 / o4-mini / 5.1 / 5.2, Claude Haiku 4.5 / Opus 4.5) and open (GPT-OSS-20B/120B, Qwen3 235B, Qwen-2.5-Coder-7B-Instruct, Kimi K2 Instruct, DeepSeek-V3.1-Terminus).

CaP-Agent0 layers three mechanisms on a frontier base model: (1) the VDM grounds each observation in structured text rather than raw pixels, mitigating the M2 cross-modal alignment failure; (2) the persistent skill library is bootstrapped by extracting via regex the recurring function definitions from successful S3 trajectories, having an LLM identify task-agnostic logic, and promoting it as reusable skills (9 skills are currently in the library) — distinguishing the approach from fixed human-designed APIs because the skills are discovered and retain the expressivity of low-level interfaces; (3) parallel reasoning concurrently samples either 9 candidates from one model or 3 each from three models with varied temperatures, then a central coding agent synthesizes the final program. CaP-RL runs GRPO with verifiable environment rewards directly on a Qwen-2.5-Coder-7B coding agent operating on the same primitive set, with policies transferring zero-shot to the real Franka.

Results

CaP-Bench frontier results in single-turn S4: no model matches human-expert Pass@1, with closed-source models leading and newer models stronger [§3.3, Fig. 1]. Single-turn primitive abstraction sweep (S4 → S1): monotone improvement, steepest for weaker / open models whose compilation rates collapse at S3/S4 [§3.3, Fig. 3]. Multi-turn additions: +M1 (text-only stdout/stderr) lifts all models; +M2 (raw RGB interleaved) degrades vs M1; +M3 (VDM) substantially beats both [Fig. 5]. M4 (low-level + multi-turn + VDM) reaches M3 parity [Fig. 6].

CaP-Agent0 (Gemini-3-Pro base) matches or exceeds human-expert programs on 4 of 7 core CaP-Bench tasks, evaluated over 100 trials per task [§4.1, Fig. 8]. LIBERO-PRO (30 manipulation tasks): VLAs OpenVLA and π0 score 0% under both Pos and Task perturbations across libero-{object, goal, spatial}; π0.5 reaches 0.17 / 0.38 / 0.20 (Pos) but ≤0.01 on Task; CaP-Agent0 reaches 0.22 / 0.26 / 0.12 (Pos) and 0.18 / 0.17 / 0.14 (Task) [Table 2]. BEHAVIOR mobile manipulation: CaP-Agent0 nav 80% / 84%, task 56% / 72% on Radio / Soda Can, vs S3 at 72% / 52% nav and 24% / 32% task, with the gain attributed to repositioning for better view and resampling grasp poses after disturbances [Table 3].

CaP-RL on Qwen-2.5-Coder-7B: 20% → 72% average success in simulation over 50 iterations; transfers to real Franka at 84% (cube lift) / 76% (cube stack), approaching human expert performance [§5, project page].

Why it’s interesting

This is the cleanest filed instance to date of “embodied RL environment as a packaged, distributable artifact” — CaP-Gym + CaP-Bench is structurally the robotics-domain entry to the RL Environment Platforms cluster that previously held SETA: Scaling Environments for Terminal Agents (env repo) (terminal), Toolathlon-GYM: Large-Scale Long-Horizon Environments for Tool-Use Agents (enterprise workflows), and OpenReward — 330+ RL Environments through one API (General Reasoning announcement) (general API). Same architectural shape (per-task primitive bundle, stateless services for parallel eval, verifiable rewards via on-policy GRPO over the coding agent), with the action space being IK/motion-planner calls and SAM3/Molmo2 perception calls instead of shell or MCP. The “test-time compute scaling hypothesis” framing — that multi-turn iteration + structured visual grounding compensates for operating over low-level primitives — is the same scaling axis as MiroThinker’s “interaction depth” in Inference-Time Scaling, now grounded in embodied control rather than web research. The auto-synthesized skill library is also structurally close to the multi-agent role-playing synthesis pipelines in Tool-Use Agents (InfTool, MiroThinker) — successful trajectories distill into reusable code primitives, with the closed loop between rollout and skill discovery emerging from the same execution-grounded verification primitive. The headline VLA comparison (OpenVLA / π0 at 0% on LIBERO-PRO under task perturbations vs CaP-Agent0 at 0.14–0.18 training-free) is a strong datapoint for the Code-as-Policy paradigm as a complement to data-intensive VLA training, especially on instruction generalization.