Raven Part 1 — Memory as a Set of Slots

TL;DR

Raven is a recurrent sequence model that fixes the recall failures of SSMs and sliding-window attention (SWA) by treating its hidden state as a set of independently-addressable slots and learning a sparse router that decides, per token, which slots to update and which to preserve. The framework — Router State Model (RSM) — unifies SSMs (dense router) and SWA (one-hot cyclic router) under a single state-update equation $S_t = (1 - r_t) \odot S_{t-1} + r_t \odot (D_t S_{t-1} A_t + U_t)$ , and Raven is the instantiation where $r_t$ is learned and sparse. The headline claim from the announcement tweet is that Raven surpasses all prior linear models on recall benchmarks even at 16× its training sequence length — addressing the persistence gap (SSMs decay too diffusely; SWA evicts too rigidly) that has dogged linear-time sequence models on Needle-In-The-Haystack and similar constant-memory recall tasks.

Key claims

SSMs fail on constant-memory recall tasks (e.g. Needle-In-The-Haystack) despite having ample state capacity to store the relevant key/value; the failure is a structural problem — every token’s write lands across all $M$ rows of the state via $S_t = S_{t-1} \odot a_t + v_t k_t^\top$ , so new tokens perturb everything already stored and interference accumulates [§“The problem is not a lack of space”].
SWA fails complementarily: it writes precisely (one-hot selector $e_t$ into one slot) but cannot reorganize memory — it just drops the oldest entry every step, so any item that needs to persist longer than the window is gone unconditionally [§“Sliding window attention”].
The RSM framework unifies both via $S_t = (1 - r_t) \odot S_{t-1} + r_t \odot (D_t S_{t-1} A_t + U_t)$ , where $r_t \in \mathbb{R}^M$ is a per-slot router; setting $r_t = \mathbf{1}$ recovers an SSM, setting $r_t = e_t$ (one-hot cyclic) recovers SWA [§“Bridging the gap”].
Making $r_t$ sparse and learned gives the state two properties simultaneously — persistence (slots not selected stay mathematically identical to the previous step, $r_t = 0 \Rightarrow$ “shielded”) and selective update (only active slots see the decay $D_t, A_t$ and write $U_t$ ) [§“RSM uses a learned sparse router”].
Raven instantiates the RSM with a learned sparse router and is positioned as a Mixture-of-Experts over memory slots — slot allocation is a routing problem solved per-token, with the same MoE intuitions for capacity, sparsity, and load-balancing [§Raven].
The recall axis Raven targets is constant-memory recall (e.g. NIAH, where in principle only the matched (key,value) needs to be retained), which is distinct from scaling-memory recall like KV-Retrieval where the query comes after the dictionary and $O(T)$ memory is fundamentally required [§“Consider Needle-In-The-Haystack”].
The companion announcement tweet reports Raven “surpassing all prior linear models even at 16× their training sequence length” on recall benchmarks (claim attributed to the tweet, not the Part-1 blog body itself).

Method

Raven’s per-step state update is the RSM equation $S_t = (1 - r_t) \odot S_{t-1} + r_t \odot (D_t\,S_{t-1}\,A_t + U_t)$ , written so the hidden state $S_t$ decomposes into $M$ independent subspaces (“slots”). For each slot $i$ : if $r_t[i] = 0$ , that row of the state is carried over unchanged (persistence); if $r_t[i] = 1$ , that row undergoes the SSM-style decay-and-write $(D_t, A_t, U_t)$ . The router $r_t$ is data-dependent (function of the current token’s input projection) and is trained to be sparse — i.e. most slots are inactive most of the time, mimicking the human intuition that a new arriving item only needs to overwrite one or a few existing entries in a closet, not the whole closet.

This casts memory allocation as a Mixture-of-Experts problem with the slots as experts, the router selecting which “experts” (slots) get updated by the current token, and the unselected slots’ contents preserved verbatim. Part 1 of the series establishes the framework and the unifying view; Part 2 (forthcoming) covers Raven’s specific block architecture, training-time design decisions, and the length-extrapolation results that motivate the 16× claim. The framework sits in the lineage of Goomba Lab’s prior work on SSM recall failures (e.g. Gather-and-Aggregate heads in Mamba-3: Improved Sequence Modeling using State Space Principles and the retrieval-aware distillation line by the same first author).

Results

Part 1 of the blog series is framework-only — it does not include the benchmark tables yet. The accompanying tweet from @avivbick (May 7, 2026) is the primary results source: Raven “surpasses all prior linear models” on recall-heavy tasks “even at 16× their training sequence length.” A companion tweet (@Xinyu2ML, quoted alongside) describes Raven as “the first SSM with selective memory allocation” achieving “SOTA performance on recall-heavy tasks with the highest length [generalization].” Quantitative numbers, exact baselines, and per-task breakdowns are deferred to Part 2 / the paper.

Why it’s interesting

Raven is the direct continuation of the Goomba Lab line that produced Mamba-3: Improved Sequence Modeling using State Space Principles (Mamba-3, with Aviv Bick as a co-author) and the retrieval-aware distillation paper. Where Mamba-3 raised the general quality of fixed-state-size linear models and explicitly recommended hybrid Mamba-3 + global-attention stacks to close the retrieval gap, Raven proposes a way to close that gap inside the linear core itself — by making memory allocation explicit (learned sparse routing over slots) rather than implicit (uniform decay across all rows). This is conceptually close to STEM: Scaling Transformers with Embedding Modules‘s argument that architectural parametric memory beats learned-routed parametric memory, except Raven applies it to the recurrent state rather than the FFN. The MoE-over-slots framing also connects to the MoE Routing Design line: many of the same questions (router sparsity, load balancing, top-k vs soft) reappear here as memory-management questions, suggesting that the MoE infra Luma already has for FFNs could inform a Raven-style state. Finally, it complements MALT Diffusion: Memory-Augmented Latent Transformers for Any-Length Video Generation‘s memory-augmented latent transformers for long video, which uses a separate compressed memory vector cross-attended into each segment — Raven’s slot-level routing is a candidate for replacing that ad-hoc memory module with a principled, learned-router state.