Guava: An Effective and Universal Harness for Embodied Manipulation

ListenGuava: A 4B Robot Brain That Beat Its Teacher10:41

Show notes

This episode digs into **Guava**, a harness framework for VLM-driven robot manipulation that ablates the design space of agent workflows, action spaces, and observations — and then distills those capabilities from a frontier VLM into a compact 4B open-source model. We cover the three load-bearing ingredients, the SFT-plus-GRPO distillation recipe on fewer than 2K simulation trajectories, and the surprising result: the small student beats its frontier teacher and transfers zero-shot to a real Franka arm. We also pull out the parallels to how we bake frontier-model behaviors into smaller generative policies. **Sources** - Guava: An Effective and Universal Harness for Embodied Manipulation

Subscribe to the channel: https://podcast.lumalabs.link/27a25f51ea81d41a8253f46c6133a35e/feed.xml

TL;DR

Guava is a harness framework for VLM-driven embodied manipulation that ablates the design space of agent workflows, action spaces, and observation spaces on six long-horizon Robosuite tasks, identifying three load-bearing ingredients: (1) iterative ReAct-style perception-reasoning-action loops, (2) semantic action abstractions delegating motion planning to lower-level controllers, and (3) multimodal (visual + textual state) observations. The paper then distills these capabilities from a frontier VLM (GPT-5.4) into Qwen3.5-4B using <2K simulation trajectories via SFT + GRPO, yielding Guava-Agent-4B which reaches 75.6% overall success across 15 in-distribution / OOD-object / OOD-prompt / long-horizon sim tasks vs 70.2% for GPT-5.4 and 62.7% for the concurrent CaP-Agent0, and transfers zero-shot to a real Franka Research 3 at 86% / 92% ID/OOD.

Key claims

Iterative ReAct workflow, semantic action abstractions, and multimodal (image + text state) observations are jointly load-bearing for GPT-5.4 under the Guava harness across six long-horizon Robosuite tasks — swapping in any of the alternatives consistently degrades performance [§3.1, Fig. 2].
The tool set combines highly abstracted primitives (grasp(object), align(object, direction, clearance)) with lower-level move(x,y,z) / rotate(angle, axis) / gripper primitives, so the VLM plans over semantic units and only falls through to metric control when necessary [Table 1].
Guava-Agent-4B achieves the highest overall sim success rate at 75.6%, ahead of GPT-5.4 (70.2%) and CaP-Agent0 (62.7%) across 15 tasks spanning ID, OOD-object, OOD-prompt, and OOD long-horizon categories [Table 2].
Zero-shot sim-to-real transfer on Franka Research 3: 86% ID and 92% OOD average across the real-world task suite, including 100% on pick up orange, place can in box, and remove cube from tray; 60% on the harder push basket vs 40% for GPT-5.4 [Fig. 4].
RL post-training with a sparse task-success reward via GRPO substantially lifts the SFT policy on long-horizon tasks: shell game 6.7% → 60.0% and place all red objects in basket 0.0% → 93.3% [Fig. 6].
Recovery behaviors on unseen execution failures — including real-world joint-limit and unreachable-pose errors — emerge from the RL-trained policy, generating plausible corrective actions like target-pose replanning or homing before retry rather than memorizing predefined correction patterns [Fig. 8].
Fewer than 2K trajectories collected entirely in simulation, augmented with recovery trajectories generated from perturbed execution states, are sufficient to transfer frontier-VLM embodied capabilities to a compact 4B open-source model [§3.2].
Named limitations: no dexterous manipulation (current action primitives are non-dexterous), the system cannot directly correct tool-level errors (invalid grasp proposals, incorrect SAM3 segmentations) though it can retry, and the setup assumes a single fixed camera view [§5].

Method

Guava wraps a VLM in a closed-loop ReAct agent over a fixed toolset of semantic action primitives (see Table 1: grasp, align, get_position, get_position_size, move, rotate, close_gripper, release, home_pose). Each observation combines a rendered RGB image with a compact textual state description (robot proprioception + task progress). The VLM emits one tool call per turn; the tool executes via lower-level motion planners and returns a new multimodal observation, closing the loop. The design ablations in §3.1 evaluate GPT-5.4 under this harness against variants that (i) collapse the ReAct loop into single-shot planning, (ii) replace semantic primitives with joint-space controls, or (iii) drop the visual or textual observation channel — the full configuration consistently wins.

For the distillation stage (§3.2), Guava is used as a data engine: frontier VLMs under the harness generate ReAct trajectories on manipulation tasks, and successful trajectories are augmented with recovery trajectories synthesized from perturbed execution states. The compact model (Qwen3.5-4B, vision encoder and aligner frozen) is post-trained in two stages: SFT on the combined success + recovery corpus, then GRPO with a sparse task-success reward on more challenging long-horizon tasks. All training uses 8× H100 in bfloat16.

Results

Overall sim success across 15 tasks: Guava-Agent-4B 75.6% > GPT-5.4 70.2% > CaP-Agent0 62.7% > Qwen3.5-4B base (no post-training) 23.1% [Table 2].
ID tasks: 100% on place can in box and remove cube from tray; 60% on the pushing task push basket vs GPT-5.4’s 26.7% [Table 2].
OOD object: 100% on pick up carrot and lemon in bin; 73.3% on push pot vs GPT-5.4’s 26.7% (largest OOD-object margin) [Table 2].
OOD prompt: 100% on stack cube reverse order; 66.7% on arrange by size reverse direction [Table 2].
OOD long-horizon: 93.3% on separate food and utensils and set table (ties with GPT-5.4). Two failure cases: shell game (6.7% after SFT → 60.0% after GRPO) and place all red objects in basket (0.0% after SFT → 93.3% after GRPO) [Table 2, Fig. 6].
Real-world Franka Research 3: 86% overall ID, 92% overall OOD; perfect scores on pick up orange, place can in box, remove cube from tray; 100% on move object away and 90% on set table OOD [Fig. 4].
Recovery emerges beyond training distribution — the agent generates plausible corrective actions for unseen real-world joint-limit and unreachable-pose failures rather than memorizing correction patterns [Fig. 8].

Why it’s interesting

Guava sits squarely on the LLM/VLM-as-controller row of the recipe-lever board that Show-Harness: Just a VLM Agent Can Play Robots Show-Harness and Agent as Policy for Robotic Manipulation AGP have been staking out, with a distinct interface bet — high-level semantic-action tool calls (grasp(object), align(...)) with an escape hatch to metric control — versus Show-Harness’s discrete named action units (MV_LEFT, GRASP) with embodiment-specific interpreters. The three-ingredient ablation (iterative loop, semantic actions, multimodal observations) is a direct empirical answer to the question Tool-Use Agents has been circling: how much of LLM-as-controller success is about the loop shape, the action-space grain, and the observation channel — and lands closer to “all three are load-bearing, none is optional.”

The distillation half is the sharper contribution for VLA Models and Reasoning RL: it shows that <2K frontier-VLM ReAct trajectories plus explicit recovery-data augmentation, followed by SFT and GRPO with sparse task-success reward, is enough to transfer harness-driven embodied capability into a 4B open-source model that beats its own teacher (75.6% vs GPT-5.4’s 70.2%). This is a direct parallel to Show-Harness’s Qwen3.5-2B fine-tune matching zero-shot frontier VLMs, but with an explicit long-horizon RL stage as the mechanism for the shell-game and red-objects gains — sibling to CaP-X’s code-as-policy distillation and to Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents Harness VLA’s frozen-VLA-under-planner recipe on the “how to bake frontier-model behaviors into small policies” axis. The 92% real-world OOD success on a Franka after sim-only training is the strongest published sim-to-real generalization for a harness-distilled policy at this parameter scale.