2AM: Grounding Agent-Side Memory as Guidance for Steerable Action Models in Long-Horizon Manipulation

TL;DR

2AM isolates the “who holds task memory” question in long-horizon manipulation by giving a multimodal Agent sole possession of task memory and reducing the Action Model to a single RGB-based, episodically stateless VLA. The Agent compiles interaction history into subtask language plus optional 2D grasp / place / move hints at different time scales; the VLA is trained on structured hint labels under condition dropout, spatial noise, and temporal jitter so it tolerates imperfect Agent outputs. On LIBERO-Mem the recipe reaches 76.3% average completion vs 14.8% for the strongest reported baseline — a 61.5-point gain — arguing that Action Model capability depends not only on what the policy learned but on how precisely the Agent can steer it.

Key claims

Long-horizon manipulation memory can stay entirely Agent-side; the Action Model can be RGB-based and episodically stateless without depth, online geometry, or planner-based object motion [Abstract].
The Agent–VLA interface uses subtask language plus optional 2D grasp, place, and move hints as steering signals of varying time scale, deliberately trading tool breadth for interface bandwidth [Abstract].
The VLA is taught to be steerable via augmented demonstrations with structured hint labels, trained under condition dropout, spatial noise, and temporal jitter so it tolerates imperfect Agent outputs [Abstract].
On LIBERO-Mem, 2AM reaches 76.3% average completion vs 14.8% for the strongest reported baseline (+61.5 pp), 63.0% relaxed success, and 11.8% strict success [Abstract].
Current agentic-system evaluations confound attribution because gains may come from richer observations or alternative motor tools while failures may stem from policy or under-specified language interface; the constrained 2AM design isolates the interface variable [Abstract].

Method

2AM factors long-horizon manipulation into two components with a deliberately narrow contract. A multimodal Agent is the sole holder of task memory: it compiles interaction history into subtask language and, optionally, 2D grasp / place / move hints that bind physical intention at different time scales. A single RGB-based, episodically stateless Action Model consumes the current observation plus the Agent’s language + hint outputs and executes task-relevant motion. There is no depth stream, no online geometry, and no planner-based object motion in the pipeline.

To make the VLA robust to noisy Agent outputs, demonstrations are augmented with structured hint labels and the policy is trained with three regularizers matched to the failure modes of an LLM/VLM steering interface: condition dropout (some hints missing), spatial noise (hints slightly off), and temporal jitter (hints arriving early or late). At inference the Agent invokes the VLA at subtask cadence and passes whatever hints it can generate; the VLA’s condition-dropout training lets it fall back on language alone when spatial hints are omitted.

Results

On LIBERO-Mem, without depth, online geometry, or planner-based object motion, 2AM reaches 76.3% average completion, a +61.5-point improvement over the strongest reported baseline of 14.8%. On the same benchmark it reports 63.0% relaxed success and 11.8% strict success [Abstract]. The paper does not disclose the identity of the strongest prior baseline or per-task numbers in the abstract; the load-bearing framing is that the gain comes from where memory lives and how precisely the Agent steers the VLA, not from adding tools or observation channels.

Why it’s interesting

2AM slots into a growing “wrap frozen (or narrow) VLA in a memory-holding orchestrator” cluster on the wiki — Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents (Harness VLA: LLM planner exposes π0.5 as a retryable contact primitive), EmbodiedSkills: A Unified Framework for Orchestrating, Training, and Deploying VLA Agents (EmbodiedSkills: shared executable-skill interface), RoboHarness: Memory-Driven Orchestration of Heterogeneous Robot Policies for Long-Horizon Planning (RoboHarness: memory bridge across heterogeneous policies), and Learning What to Say to Your VLA: Mostly Harmless VLA Steering (LFP: test-time language-rewrite steering). 2AM’s contribution to this cluster is deliberate scope reduction: instead of expanding the tool library, it argues that widening the Agent→VLA interface bandwidth (structured 2D hints at multiple time scales, learned as first-class conditioning) with a stateless RGB-only executor is enough to unlock long-horizon memory tasks. The +61.5-point margin on LIBERO-Mem — a memory-dependent benchmark where EmbodiedSkills’ own agent loop reports only 12.5% on four RMBench memory tasks — is the sharpest datapoint yet that the interface between planner and policy, not orchestrator sophistication, is the missing lever for the memory-dependent regime.