MEM: Multi-Scale Embodied Memory for Vision Language Action Models

TL;DR

MEM (Physical Intelligence + Stanford/Berkeley/MIT) equips π₀-style VLAs with two memory scales: a dense video-encoder short-term memory covering tens of seconds, and a language-based long-term memory in which the high-level policy compresses semantic events into a running natural-language summary. Instantiated as π-MEM on Gemma3-4B, this enables robot policies to execute 15-minute long-horizon tasks (recipe setup, cleaning a full kitchen) and unlocks in-context adaptation — the policy watches its own failed grasp attempts and modifies its manipulation strategy on the fly. Every ablation confirms both memory scales are essential: removing video memory breaks fine-grained continuation, removing compressed language memory destroys progress tracking, and using naïve concatenated subtask instructions instead of compressed summaries causes severe train-inference distribution shift.

Key claims

The high-level policy is factorized into p(a | context) = ∫ p_low(a | g, o_recent, i) · p_high(i, m' | g, o_recent, m), where m is the language memory and the high-level policy explicitly predicts its own updated memory m' at each step [§III-A].
Language-memory training data is generated by prompting an off-the-shelf LLM to summarize prior subtask annotations while actively compressing/discarding obsolete detail, which reduces train-inference distribution shift versus naïve concatenation [§III-B].
The video encoder factorizes attention into standard spatial attention plus a causal temporal attention every 4th ViT layer, reducing joint attention cost from O((T·N)²) to O(T·N² + T²·N), and drops all past-timestep tokens after the encoder — matching single-image VLA token counts downstream [§III-C].
The video encoder adds zero new learnable parameters over a standard ViT (temporal position is a fixed sinusoid initialized so the T=1 case exactly matches a pretrained VLM’s ViT), allowing initialization from any pretrained VLM [§III-C].
π-MEM pretraining uses 6-frame observation memory at 1-second stride and expands flexibly to 18 frames × 54 seconds at post-training time [§III-D].
π-MEM significantly outperforms memoryless π baselines on the 15-minute Recipe Setup and Clean-Up-Kitchen tasks; ablating either memory scale substantially degrades performance [Fig. 4].
Naïve language memory (concatenated subtask instructions up to a max length) performs significantly worse than model-predicted compressed summaries, because near-optimal demonstrations utter each subtask once while inference-time policies repeat failing subtasks, creating a distribution shift [§IV-A].
With correction data in short-term memory during finetuning, the policy learns in-context adaptation — modifying grasp height after a mis-grasp of chopsticks and trying the opposite fridge-door direction after a failed opening — while the memoryless baseline cannot leverage the same correction data [§IV-B, Fig. 5].
On a benchmark of memory-requiring tasks (partial observability, counting, timing, spatial memory across single-arm/dual-arm/mobile embodiments), MEM outperforms observation-compression baselines (Pool Memory-style) [§IV-C].

Method

Two memory representations, both trained jointly. Short-term visual memory: a modified ViT patchifies each frame independently, applies standard spatial attention, and every 4th layer additively adds a causal temporal attention over corresponding patches across time. Only the current timestep’s patch representation is passed downstream, so the VLA backbone sees the same token count as a single-image VLA — the temporal information is compressed into the current-frame tokens. Weights initialize directly from any pretrained VLM’s ViT (sinusoidal temporal PE = 0 at T=1). Long-term language memory: the high-level policy consumes (goal, current observation, previous memory m) and emits (subtask instruction i, updated memory m'). Training data for m' | m, i, success/failure is synthesized by prompting an off-the-shelf LLM to write compressed summaries — actively removing detail no longer needed (three specific bowls become “three bowls in top-right cabinet”). π-MEM integrates this into π (Gemma3-4B backbone, FAST discrete tokens + 860M flow-matching action expert, gradients don’t flow from expert to backbone). Proprioceptive state history is projected through a linear layer into backbone embedding space to avoid ballooning text-token count. Real-time chunking (RTC / train-time RTC) is used at deployment for asynchronous inference.

Results

15-minute Recipe Setup and Clean-Up-Kitchen tasks: π-MEM substantially improves over memoryless π; ablating either video memory or language memory drops success and progress scores meaningfully; ablating the compression on language memory (naïve concatenation) also degrades performance [Fig. 4].
In-context adaptation on chopstick-picking (out-of-distribution table height causing mis-grasps) and fridge-opening (unknown open direction): with correction data trained into short-term memory, π-MEM adapts grasp height and reverses opening direction across trials; memoryless baseline shows no benefit from the same correction data [Fig. 5].
On a diverse memory-benchmark suite covering partial observability, counting scoops of coffee, timing a grilled cheese, and spatial memory over cleaned window regions, MEM outperforms Pool-Memory-style observation compression baselines [§IV-C, Figs. 6 & 8].

Why it’s interesting

MEM lands in the memory-augmented VLA design space alongside RoboMemArena: A Comprehensive and Challenging Robotic Memory Benchmark (which is the benchmark side of the same question), and it’s the second concrete instance filed here (after ABot-AgentOS: A General Robotic Agent OS with Lifelong Multi-modal Memory) of the two-timescale split — short-horizon dense observations + long-horizon symbolic/lexical memory. The video-encoder trick is architecturally elegant: no new parameters, same downstream token count, and by initialization it degenerates to the pretrained ViT at T=1, so a memoryless VLA can be “grown” into a memory-VLA without retraining the backbone. The naïve-concatenation ablation is the paper’s most useful negative result — it’s the same distribution-shift concern that pushed the compression design choice, and it suggests other memory approaches that treat memory as a concatenated log (ABot-AgentOS: A General Robotic Agent OS with Lifelong Multi-modal Memory) will need equivalent compression discipline to hold up under repeated-failure inference-time behavior. The in-context adaptation demonstration is a genuinely new capability window: correction data now teaches the policy when to change strategy, not just what correct behavior looks like.