Skip to content
Bud's Wiki

RoboHarness: Memory-Driven Orchestration of Heterogeneous Robot Policies for Long-Horizon Planning

TL;DR

Section titled “TL;DR”

RoboHarness (Huawei Noah’s Ark; Huang et al.) is a general orchestration framework that treats independently developed robot control systems — VLAs, RL policies, and task-and-motion planning (TAMP) — as reusable agentic skills with uncertain capability boundaries. The load-bearing move is a Memory Bridge: on each policy handoff the harness retrieves execution trajectories associated with the next policy, estimates its in-distribution state region, and steers the robot toward that region before invoking it — so heterogeneous policies compose without joint retraining. Multi-modal execution memory and online evidence characterize each policy’s operating range for capability-aware decomposition and routing. Reports substantial gains in zero-shot long-horizon planning and OOD robustness across three public benchmarks, 500 customized tasks, and 135 real-robot experiments.

Key claims

Section titled “Key claims”
  • Existing planning methods built on homogeneous, predefined skills with fixed applicability overlook two problems that arise when composing heterogeneous policies: uncertain capability boundaries and cross-policy distribution mismatch on handoff [Abstract].
  • RoboHarness encapsulates independently developed robot control systems (VLAs, RL policies, TAMP) as reusable agentic skills within one unified framework, with the framework designed to also accept navigation policies, MPC controllers, and world-action models [Abstract].
  • Multi-modal execution memory + online evidence are used to characterize each policy’s capability boundary, which then drives both capability-aware task decomposition and per-subtask routing [Abstract].
  • The Memory Bridge stabilizes policy handoffs by retrieving execution trajectories associated with the next policy, estimating its in-distribution state region, and guiding the robot toward that region — without any joint policy retraining [Abstract].
  • Extensive experiments on three public benchmarks + 500 customized tasks + 135 real-robot experiments demonstrate effective capability-aware routing and stable orchestration, with substantial improvements in zero-shot long-horizon planning and OOD robustness (no headline numbers in the abstract) [Abstract].

Method

Section titled “Method”

RoboHarness is a two-level system. The upper level is a planner that (a) decomposes a long-horizon task into subgoals and (b) routes each subgoal to one of the wrapped policies (VLA, RL, TAMP, or in principle MPC / navigation / WAM). Routing decisions consult a multi-modal execution memory that stores per-policy trajectories plus online evidence, from which the harness estimates each policy’s capability boundary — the region of state space where it is empirically reliable.

The lower level is the Memory Bridge, which runs between consecutive policy invocations. Before invoking policy B after policy A finishes, the bridge retrieves trajectories from memory that are associated with B, estimates the in-distribution state region for B, and guides the robot (with A or an intermediate primitive) toward that region so B is invoked inside its known-good regime. This is what removes the requirement to jointly retrain policies to make them compose — each policy stays frozen, and the bridge absorbs the cross-policy distribution mismatch. The abstract does not disclose the concrete implementation of the state-region estimator or the guiding controller; those details will need a targeted refresh.

Results

Section titled “Results”
  • Three public benchmarks + 500 customized tasks + 135 real-robot experiments [Abstract].
  • Reports “substantial improvements in zero-shot long-horizon planning and out-of-distribution robustness” [Abstract]. The abstract does not disclose per-benchmark numbers, ablations of Memory Bridge vs. no-bridge routing, or breakdowns by policy type (VLA / RL / TAMP) — refresh needed once the body is retrievable.
  • Real-robot count (135 experiments) is notable at filing: comparable to or larger than Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents‘s hardware evaluation.

Why it’s interesting

Section titled “Why it’s interesting”

Second filed paper this week (with Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents) that answers the “wrap the frozen policy in an orchestrator” question — and the two occupy complementary points in the design space. Harness VLA takes one policy type (a frozen VLA, π0.5) and expands the analytic primitive library around it, learning where the VLA is reliable versus where analytic primitives should take over. RoboHarness inverts the framing: it accepts arbitrarily many heterogeneous learned policies (VLAs, RL, TAMP) and asks how to route between them when their in-distribution regions don’t align — which is why Memory Bridge is the load-bearing contribution here (Harness VLA doesn’t need one because its non-VLA primitives are analytic and have well-defined preconditions). This is a stricter generalization of VoLo: A Physical Orchestrator for Open-Vocabulary Long-Horizon Manipulation‘s Physical Orchestration and Learning What to Say to Your VLA: Mostly Harmless Vision Language Action Model Steering‘s LFP formalization, and pushes further into the multi-policy regime that GaP: A Graph-as-Policy Multi-Agent Self-Learning Harness For Variational Automation Tasks‘s Graph-as-Policy also inhabits — though GaP builds one composed graph over a homogeneous 51-skill library, while RoboHarness treats each policy family as a black-box skill and worries about the handoff. The Slack thread’s read is right: “more and more people bet in this direction” — the design space is converging on capability-aware routing + memory-driven handoff stabilization as the substrate for long-horizon embodied tool-use, with heterogeneity as the axis this paper adds.

See also

Section titled “See also”