Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents

TL;DR

Harness VLA (RLinf; Zhang et al.) proposes a memory-augmented agentic harness that exposes a frozen VLA (e.g. π0.5) as a retryable contact-rich primitive alongside a small fixed library of analytic primitives (grounding, staging, transport, navigation, release). Rather than expand the skill library, the harness learns the operating range of each fixed primitive from task-specific execution traces, global success rules, and failure models — an LLM planner absorbs the semantic re-grounding, non-contact execution, and VLA re-staging while the VLA is reserved for local contact-rich phases. On perturbed tabletop / kitchen / bimanual manipulation, Harness VLA beats the strongest baselines by +38.6 pp on LIBERO-Pro and +25.4 pp on RoboCasa365, and reaches 58.4% on RoboTwin C2R — extending pretrained VLAs beyond their training distribution without any finetuning.

Key claims

End-to-end VLAs fail under deployment perturbations — semantic retargeting, goal re-binding, spatial-layout shifts, unstable local contacts — because they are trained on in-distribution task trajectories, while LLM coding agents provide complementary semantic/compositional reasoning but their analytic primitives struggle with irregular grasping, constrained placement, and articulated-object interaction [Abstract].
The core recipe is learning the operating range of a fixed primitive library, not expanding it. Task-specific execution traces + global success rules + failure models are compiled into memory that tells the planner where each analytic primitive (and the frozen VLA) is reliable [Abstract].
The VLA is reserved for local contact-rich phases only — the planner lifts semantic re-grounding, non-contact execution (transport / navigation), and VLA re-staging out of the VLA and into analytic primitives; the VLA is called as a retryable contact primitive [Abstract].
On LIBERO-Pro, Harness VLA beats the strongest relevant baseline by +38.6 percentage points; on RoboCasa365 by +25.4 pp; on RoboTwin C2R (clean-to-randomized bimanual) reaches 58.4% [Abstract].
No VLA finetuning is required — the harness extends pretrained VLAs beyond their original trajectory distribution purely through planner + memory + retry [Abstract].

Method

The harness is an LLM planner + memory system that wraps a frozen VLA (π0.5 in the released code) plus a fixed library of analytic primitives — grounding, staging, transport, navigation, release. The planner is responsible for the composition and the staging around each VLA call, while the VLA itself is exposed as a narrow API of retryable contact-rich primitives (the released code implements pi0_pick and pi0_doubled as the callable VLA-side entry points, per the Slack thread’s code review). Steering is done via natural-language prompt engineering — a perception system prompt plus per-task hybrid guides that decide when the VLA should begin acting (e.g. “π0.5 only does the grasp”) and when analytic primitives take over. Memory captures three complementary signals: task-specific execution traces (what happened), global success rules (what “done” looks like), and failure models (what went wrong last time), which are consulted at planning time so that the operating range of each primitive tightens with experience.

Results

LIBERO-Pro: +38.6 percentage points over the strongest relevant baseline [Abstract].
RoboCasa365: +25.4 percentage points over the strongest relevant baseline [Abstract].
RoboTwin C2R (clean-to-randomized bimanual): 58.4% [Abstract].
All numbers are reported without any VLA finetuning; the underlying VLA (π0.5) is frozen throughout [Abstract].
Ablation-level detail (which of grounding / staging / transport / navigation / release the harness contributes most, how much memory helps vs. a memory-less planner, whether steering degrades gracefully as the prompt library shrinks) is not visible from the abstract alone; the paper body will need a targeted refresh.

Why it’s interesting

This is the sharpest empirical instance yet of the VLA Models “wrap the frozen VLA in an orchestrator” recipe that VoLo: A Physical Orchestrator for Open-Vocabulary Long-Horizon Manipulation introduced as Physical Orchestration and that Learning What to Say to Your VLA: Mostly Harmless Vision Language Action Model Steering formalized as Language Feedback Policy — Harness VLA reports +38.6 pp on LIBERO-Pro, the same suite where CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation observed OpenVLA and π0 collapse to 0% under position and instruction perturbations. Unlike VoLo, which frames the VLA as an interruptible mid-rollout tool, Harness VLA takes a stricter position: the VLA is only invoked for the contact-rich sub-phase (the released strict_hybrid_guide restricts π0.5 to pi0_pick), while analytic primitives handle everything the VLA is unreliable at. It also sits in the Tool-Use Agents design space as a domain-specialized ReAct-like loop where the “tools” are typed physical primitives and a narrow VLA API, sibling in spirit to RoboClaw: An Agentic Framework for Scalable Long-Horizon Robotic Tasks‘s MCP tool interface but pushed further toward the CaP-X (CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation) and GaP (GaP: A Graph-as-Policy Multi-Agent Self-Learning Harness For Variational Automation Tasks) view that a small verified primitive library plus structural composition can outperform end-to-end scaling on perturbed evaluation — with the concrete headline that no VLA finetuning is required for a +25–38 pp jump.