Skip to content
Bud's Wiki

RoboMemArena: A Comprehensive and Challenging Robotic Memory Benchmark

TL;DR

Section titled “TL;DR”

RoboMemArena is a large-scale robotic-memory benchmark of 26 simulation tasks with average trajectory length >1,000 steps and 68.9% memory-dependent subtasks, plus 5 paired real-world tasks on the AgileX Cobot Mobile Aloha platform. The generation pipeline uses a VLM to decompose long-horizon tasks into atomic subgoals, executes them autonomously via AnyGrasp with retry-on-failure, and extracts multimodal keyframe annotations from gripper-state transitions and end-effector kinematic inflections. The authors also release PrediMem, a dual-system VLA (Qwen3-VL-8B S2 planner + π0.5-style S1 actor) with a keyframe bank + recent-frame buffer and a predictive-coding auxiliary head that trains the S2 to anticipate the next visual latent, reporting 38.5% TSR / 55.2% CSR — a large jump over π0.5 (21.5/38.7) and MemER (27.3/49.1) — and showing closed-source GPT-5.4 as an out-of-the-box S2 collapses to 8.7% TSR, arguing memory-management VLMs need robotic-domain training.

Key claims

Section titled “Key claims”
  • The benchmark’s 68.9% memory-dependent-subtask ratio (104 of 151 subtasks) is the highest among filed robotic memory benchmarks, and its 1,076-step average trajectory length beats prior memory-focused benchmarks [§3.3, Fig. 2(a,c)].
  • Four memory-demand categories (Multi-Object Transferring, Multi-Object Occlusion, Multi-Object Counting, Multi-Object Sequence) cover the failure modes of reactive VLAs; Occlusion is the largest category with 11 tasks, reflecting how often visually-closed containers break reactive policies in household settings [§3.1].
  • Data-generation pipeline resolves the automatic-collection vs fine-grained-annotation trade-off in three stages: VLM-driven task decomposition, AnyGrasp-based autonomous execution with post-condition retry, and multi-conditioned keyframe extraction that unions gripper-state transitions with end-effector kinematic inflections [§3.2, Eq. 1-3].
  • Reactive π0.5 reaches only 21.5% average TSR / 38.7% CSR because it lacks history modeling, failing almost completely on drawer-based occlusion tasks by repeatedly reopening drawers it has already inspected [§5.2, Table 2(a)].
  • Prior memory-augmented baselines are limited: HiF-VLA does not explicitly store task-level events (16.9% TSR), MemoryVLA’s token-level memory is not aligned with sparse physical transitions (15.0% TSR), MemER’s dual-system keyframe retrieval outperforms reactive baselines but its S2 VLM cannot select informative keyframes without task-dynamics training (27.3% TSR) [§5.2, Table 2(a)].
  • PrediMem combines an explicit keyframe bank + recent-frame sliding window with a predictive-coding auxiliary head (latent MSE + cosine loss against a frozen ViT teacher’s next-frame features, weight 0.1) and reaches 38.5% TSR / 55.2% CSR — the best filed number on the benchmark [§5.2, Table 2(a); §5.3, Table 3].
  • The predictive-coding head is a training-only auxiliary that is removed at inference, so PrediMem retains standard dual-system runtime cost while inheriting improved keyframe sensitivity [§4, Inference paragraph].
  • Ablation: removing the predictive-coding head drops TSR from 38.5% → 32.3%; removing the keyframe bank drops it further to 17.7%, and both are essential for occlusion, counting, and sequence tasks where correct decisions depend on events outside the recent window [§5.3, Table 2(c)].
  • Predictive coding produces more discriminative keyframe representations: t-SNE of final-layer hidden states shows tight per-class clusters with predictive coding vs strong overlap without [§5.3, Fig. 4(c)].
  • Frontier closed-source VLMs used zero-shot as S2 fail badly on robotic memory tasks: GPT-5.4 reaches only 8.7% TSR (vs 46.1% for ground-truth S2), suggesting VLMs must be trained on robotic data before serving as memory managers [§5.2, Table 2(b)].
  • Backbone scaling matters: PrediMem with Qwen3-1.7B backbone gets 19.9% TSR, Qwen3-4B 31.9%, Qwen3-VL-8B 38.5% — establishing a rough scaling curve for memory-management VLMs [§5.2, Table 2(d)].
  • Memory-bank scaling: recent-frame buffer of 3–5 frames is sufficient; larger windows add redundant context and slow VLM inference, breaking the async S1/S2 synchronization [§5.4, Fig. 4(a,b)].

Method

Section titled “Method”

RoboMemArena bundles simulation + real-world evaluation of embodied memory. On the data-generation side, each task starts as a high-level instruction plus RGB observation; a VLM proposes an ordered subtask sequence (manually refined where inconsistent), AnyGrasp handles per-subtask 6-DoF grasp execution through predefined primitives with post-condition-checked retry, and keyframes are extracted as the union of gripper-open/close transitions and end-effector velocity minima or direction-change points (cosine < τ). The result: 2,600 successful demonstrations across 26 tasks producing 15,100 keyframe-aligned short segments. Evaluation reports both binary Task Success Rate (all stage predicates satisfied) and Cumulative Success Rate (fraction of stages completed) to distinguish partial progress from failure.

PrediMem is a dual-system VLA. The S2 planner is Qwen3-VL-8B-Instruct with the ViT frozen; it consumes the current observation plus a memory bank M_t = M_r ∪ M_k (recent sliding window of 5 frames + uncapped keyframe buffer), predicts the current subtask, and decides whether to write the current frame into the keyframe buffer. The S1 actor is π0.5-shaped and predicts the freshest subtask-conditioned action chunk under the flow-matching objective. The key training trick is an auxiliary predictive-coding head that takes S2’s hidden state at frame t and predicts the ViT features of frame t+1, supervised by latent MSE + cosine distance against the frozen ViT (stop-gradient teacher, weight α=0.1). This reshapes S2’s hidden space to be more sensitive to physical state transitions — so keyframe decisions can be made through the standard LM head without a separate retrieval module. At inference the predictive-coding head is discarded.

Results

Section titled “Results”
  • Main table: PrediMem 38.5% TSR / 55.2% CSR vs MemER 27.3/49.1, π0.5 21.5/38.7, MemoryVLA 15.0/35.3, HiF-VLA 16.9/39.8; Ground-Truth S2 (oracle) reaches 46.1/64.8 [Table 2(a,b)].
  • Predictive-coding loss-weight sweep: α = 0.0 → 32.3%, α = 0.1 → 38.5% (best), α = 0.5 → 31.0%, α = 1.0 → 29.8% [Table 3].
  • Component ablation: without predictive-coding head 32.3% (biggest drops on occlusion/counting/sequence); without keyframe bank 17.7% (broad regression) [Table 2(c)].
  • Backbone ablation: Qwen3-1.7B 19.9%, Qwen3-4B 31.9%, Qwen3-VL-8B 38.5% — clean scaling in the memory-management VLM [Table 2(d)].
  • Closed-source VLM as S2: Qwen3-VL-8B frozen 6.0% TSR, GPT-5.4 8.7% — both far below trainable baselines, indicating multimodal-pretraining alone doesn’t transfer to robotic memory tasks [Table 2(b)].
  • Task-category breakdown: PrediMem’s largest lift is on Occlusion (7.3 → 27.3% TSR) and Counting (14.3 → 45.7% TSR) — the categories where memory-dependent decisions dominate — while Transferring shows smaller gains (20.0 → 22.5%) because state changes there are more direct [Table 2(a)].

Why it’s interesting

Section titled “Why it’s interesting”

Slots into the VLA Models recipe debate as a memory-augmented dual-system recipe distinct from the existing five levers on that concept — the predictive-coding auxiliary head sharpens the keyframe-decision signal without adding inference cost, and the frontier-VLM-fails-as-S2 result (GPT-5.4 → 8.7% TSR) is the sharpest quantitative datapoint yet that memory-management VLMs need robotic-domain training rather than raw multimodal scale. Extends the RL Environment Platforms env-as-package pattern to a memory-specific evaluation substrate — sits alongside RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies and RoboCasa365 with a distinct axis (long-horizon partial-observability rather than generalization / precision / open-vocab), and shares the paired sim+real design with RoboDojo. Concrete answer to the “how much of the VLA generalization gap is a memory-representation problem” question that In-Context World Modeling for Robotic Control (ICWM) raises with in-context system identification — where ICWM puts memory in the context window, PrediMem puts it in an explicit keyframe bank addressed by a predictive-coding-shaped hidden space, giving the field two structurally different answers to compare.

See also

Section titled “See also”