Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity

ListenSparse Delta Memory: Give the Linear State Room to Breathe9:39

Show notes

A look at Sparse Delta Memory (SDM), which scales the hidden state of gated linear RNNs by orders of magnitude using sparse addressing instead of a wider dense state. We unpack how it swaps the dense key–value outer product for sparse reads and writes into a big explicit memory, why that helps long-context recall under an iso-FLOP budget, and how learning the memory's initial state turns it into parametric memory. We also connect it to the hybrid-linear-attention debate: maybe the "how much softmax?" question is really a state-capacity question. **Sources** - Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity

Subscribe to the channel: https://podcast.lumalabs.link/27a25f51ea81d41a8253f46c6133a35e/feed.xml

TL;DR

Sparse Delta Memory (SDM) is an extension of Gated DeltaNet that scales the linear-RNN hidden state by orders of magnitude using a sparse addressing scheme — replacing the dense key–value outer-product update with sparse reads and writes into a large explicit memory. Under an iso-FLOP + iso-parameter constraint, higher explicit-state capacity lifts in-context learning and long-context retrieval performance, closing part of the recall gap that pure linear-attention primitives still concede to softmax attention. Making the memory’s initial state a learned parameter turns SDM into a parametric-memory substrate as well: the same architecture then improves on common-knowledge and reasoning benchmarks. The paper is directly relevant to the hybrid-linear-attention thread — it is a new linear primitive whose recall/FLOP curve is shaped by memory addressing rather than by state-vector width.

Key claims

Increasing the hidden-state size of gated linear RNNs improves long-context recall, but with a dense KV outer-product the compute scales unfavorably — motivating a sparse addressing scheme that decouples state capacity from per-token FLOPs [§Abstract].
SDM extends the Gated DeltaNet architecture by replacing the dense key–value outer product with sparse reads and writes into a large explicit memory [§Abstract].
Under an iso-FLOP constraint and an identical parameter count, higher state-memory capacity via SDM significantly improves performance on in-context learning and long-context retrieval tasks relative to the dense-outer-product baseline [§Abstract].
Making the initial state of the SDM memory a learned parameter turns the explicit memory into parametric memory and further improves the model on a wide range of common-knowledge and reasoning tasks [§Abstract].

Method

SDM is built on top of Gated DeltaNet. In standard delta-rule linear attention, each token forms a dense key–value outer product that additively updates a matrix state; total state capacity is bounded by the (small) hidden dimension because every update touches every state cell. SDM changes the update rule: keys are used as a sparse address into a much larger explicit memory bank, and only the addressed slots are read and written per token. This makes the memory capacity a free parameter (orders of magnitude larger than the dense-state analog) while keeping per-token compute roughly constant — the FLOPs shift into sparse gather/scatter rather than dense outer-product accumulation. When the initial state of that memory is trained along with the rest of the network, the same slots serve as parametric memory: knowledge stored across pretraining lives in the memory bank and is retrieved by the same sparse addressing at inference.

Results

The abstract reports two headline outcomes under an iso-FLOP + iso-parameter setup:

In-context learning and long-context retrieval: SDM with a larger sparse memory beats the Gated DeltaNet baseline at matched FLOPs and parameters — the recall gap that motivates hybrid linear-attention stacks (linear primitive + softmax anchors) shrinks when the linear branch is given more state capacity through sparse addressing rather than through a wider dense state.
Common-knowledge and reasoning benchmarks: learning the SDM memory’s initial state (parametric-memory mode) yields further gains on a “wide range” of these tasks.

Numerical multipliers, scales, and per-benchmark deltas are not disclosed in the abstract; the arXiv HTML body was not retrieved in this ingest.

Why it’s interesting

SDM sits at the intersection of two live threads on the wiki. First, on Hybrid Linear Attention: the current cluster is dominated by layer-wise recipes (Kimi Linear 3:1 KDA:MLA, MiniMax-M1 7:1 lightning:softmax, Qwen3-Next Gated DeltaNet + Gated Attention) that try to close the recall gap by mixing softmax anchors into a linear stack. SDM makes a different bet — keep the stack linear, but give the linear primitive orders-of-magnitude more state via sparse addressing. If it holds up under head-to-head with softmax anchors on retrieval benchmarks, it argues the ratio question (“how much softmax do you need?”) is partly downstream of a state-capacity question (“how big is your linear state?”). It also complements HydraHead: From Head-Level Functional Heterogeneity to Specialized Attention Hybridization as another orthogonal axis (head-level vs state-capacity) on which the layer-wise ratio can be reduced.

Second, on Parametric memory: SDM is the first filed instance of architectural parametric memory inside a linear-RNN state — distinct from the FFN-embedding-table lever of STEM: Scaling Transformers with Embedding Modules (which is static and outside the sequence dimension) and from the router-state slot addressing of Raven Part 1 — Memory as a Set of Slots (which routes but does not grow the memory bank). The learned-initial-state trick is close in spirit to Long context as weights: at inference, the memory is “context that lives in weights”, except the addressing is sparse and per-token rather than a full LoRA emission or a full backbone SGD step.