Skip to content
Bud's Wiki

Parametric memory

TL;DR

Section titled “TL;DR”

How facts and rare-entity knowledge are stored in network weights, and what the structure of that storage looks like — strict associative lookup, emergent geometric/relational embeddings, or explicit token-indexed embedding tables — together with the practical implication that the amount of parametric memory scales log-linearly with parameter count at ~2 bits/parameter. The filed papers cluster around three claims: (i) parametric memory in standard transformers is geometric, not just associative, and emerges from a CE-loss spectral bias (Deep sequence models tend to memorize geometrically; it is unclear why); (ii) one can make the lookup architectural by replacing the FFN up-projection with token-indexed embedding tables and recover most of the capacity benefit of MoE without learned routers (STEM: Scaling Transformers with Embedding Modules); and (iii) because factual recall is incompressible, a black-box accuracy score on rare-entity probes lower-bounds the parameter count to within ~1.5× and shows factual capacity is not compressing year-over-year (Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity). The three together justify treating parameter count as the binding resource for long-tail factual recall — even as procedural benchmarks saturate.

Key claims

Section titled “Key claims”
  • Transformers and Mamba spontaneously develop geometric embeddings over memorized graphs — non-co-occurring entities are positioned to reflect global multi-hop distances — turning L-fold compositional recall into a 1-step spatial query, and this happens at scales (10k nodes, 10-hop paths) where strict associative recall would predict failure (Deep sequence models tend to memorize geometrically; it is unclear why §2.1, Fig. 4, Fig. 6).
  • Geometric memory is not strictly more succinct than associative memory — there are graphs where both have equal parameter cost, yet the model still picks geometric — so the bias is a learning bias of CE loss, not a capacity argument; it reduces to the low-rank spectral bias already visible in shallow Node2Vec (Deep sequence models tend to memorize geometrically; it is unclear why §3.1, §5).
  • Replacing the FFN up-projection with a token-indexed embedding lookup E[token_id] (keeping gate and down dense) makes parametric memory architectural: no learned router, no all-to-all, no load-balance loss, and the giant table can live on CPU and be asynchronously prefetched. STEM beats dense baselines while removing ~1/3 of activated FFN params (STEM: Scaling Transformers with Embedding Modules §3, Abstract).
  • STEM’s learned per-token “up-projection vectors” spread out angularly (low pairwise cosine similarity), which the authors connect to reduced representational interference — i.e. parametric memory works better when the addressing scheme spreads keys (STEM: Scaling Transformers with Embedding Modules §4).
  • Storing facts requires at least ~2 bits per parameter (allenzhu2025 / morris2025memorize), so accuracy on rare-entity probes provides a Shannon-entropy lower bound on parameter count that is intrinsic to the model rather than dependent on hardware/serving assumptions (Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity §3.2).
  • IKP calibration on 89 open-weight models (135M–1.6T, 19 vendors) achieves R² = 0.917 with slope ≈0.147 (each 10× in params adds ~14.7pp accuracy on incompressible probes); LOO-CV places 68.5% of held-out open models within 1.5× of true size (Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity §6.1, §6.2).
  • For MoE models, total parameters predict knowledge capacity (R²=0.79) far better than active parameters (R²=0.51) — facts are stored across all expert weights, not just the routed ones; this is the load-bearing empirical claim for “MoE is parametric memory at total-params scale” (Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity §6.1, Fig. 2).
  • Procedural capability compresses under the Densing Law, but factual capacity does not: across 96 dated open-weight models the time coefficient on IKP accuracy is −0.05pp/month (95% CI [−0.20, +0.10]), rejecting Densing’s +1.92pp/month prediction at p<0.001 (Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity §1, Contributions §3).
  • Parametric memory has an in-state analogue distinct from weight storage: in a recurrent linear-time sequence model, the hidden state StS_t itself can be partitioned into independently-addressable slots, with a learned sparse router deciding which slots persist and which are overwritten per token — making the recall-vs-persistence trade explicit rather than emergent (Raven Part 1 — Memory as a Set of Slots).
  • In the agent-memory literature, “parametric” is one of three Forms (alongside token-level and latent) — the substrate axis on which agent memory is layered (Memory in the Age of AI Agents). The survey explicitly distinguishes the parametric-memory work studied here (continual training / model editing / distillation as memory operations) from RAG (a token-level Form) and from the LLM-internal facts studied in isolation from any agentic loop.

Recent contributions

Section titled “Recent contributions”
  • [2026-07-16] RoboMemArena: A Comprehensive and Challenging Robotic Memory Benchmark: RoboMemArena / PrediMem instantiates an external-store memory form for embodied agents (keyframe bank + recent-frame buffer read by a dual-system VLA) — distinct from the in-weights (STEM, geometric-memory), in-state (Raven, SDM), and in-context (ICWM) forms already tracked here. The novel operational lever is a training-only predictive-coding auxiliary head: supervising the S2 hidden state to predict next-frame ViT features (α=0.1 latent MSE + cosine, stop-gradient teacher) shapes representations so keyframe-write decisions can be made through the standard LM head without a retrieval module, and is discarded at inference for zero runtime cost — an analog to STEM’s “make the addressing scheme architectural” bet, but applied to the write side of an external memory store rather than the read side of a weight table.
  • [2026-07-09] Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity: SDM (Meta/FAIR) makes the initial state of a sparse-addressed linear-RNN memory bank a learned parameter — the first filed instance of architectural parametric memory inside a recurrent state. Distinct from STEM’s FFN-embedding-table lever (static, outside the sequence dimension) and Raven’s slot routing (routes a small dense state rather than growing a memory bank). Reports gains on common-knowledge and reasoning tasks when the initial state is trained end-to-end.
  • [2026-07-08] A global workspace in language models: Anthropic’s J-space (identified by a Jacobian lens on per-token output influence) is a small, densely-connected residual substructure that functions as a broadcast/workspace: a single edit (“France”→“China”) propagates across four different downstream question-answering tasks (capital/language/continent/currency), consistent with one shared representation being read by many downstream circuits rather than per-task copies. Complements longer-timescale parametric-memory work by describing an in-forward-pass shared-write substrate — the temporal analog of caching in a workspace rather than storing in weights.
  • [2026-05-25] Memory in the Age of AI Agents: Positions “parametric” as one Form among three (token-level / parametric / latent) in a broader agent-memory taxonomy that cross-products it with Functions (factual / experiential / working) and Dynamics (formation / evolution / retrieval). Reframes parametric memory as the substrate the agent-memory layer sits on rather than a free-standing object — and explicitly excludes it from “agent memory proper” when there is no agentic loop above it. The Dynamics axis (especially evolution) is the framing the prior parametric-memory work has been missing — model editing, continual training, and distillation are all “parametric-evolution” operations under this taxonomy.
  • [2026-05-24] Raven Part 1 — Memory as a Set of Slots: Raven (Goomba Lab) introduces the Router State Model (RSM) framework — a single state-update equation unifying SSMs (dense router) and SWA (one-hot cyclic router), with Raven as the learned-sparse-router instantiation. Reframes the recall failures of SSMs as a structural problem (uniform decay perturbs every stored item) rather than a capacity problem, and proposes per-slot persistence as the fix. Adds an in-state, learned-routing lever to the parametric-memory toolkit, distinct from the weight-side levers in STEM (architectural FFN replacement) and IKP (fact-density measurement).
  • [2026-05-24] Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity: Builds a black-box parameter-count estimator by treating factual recall as the lower-bound Shannon-entropy signal of parametric memory. Calibrates a log-linear acc = a + b·log10(N) curve at R²=0.917 over 89 open models, separates compressible (procedural) from incompressible (factual) capability, and shows year-over-year factual-capacity scaling has not compressed even as MMLU/HELM saturate. Headline closed-model estimates: GPT-5.5 ≈9T, Claude Opus 4.7 ≈4T, GPT-5.4 ≈2.2T, Claude Sonnet 4.6 ≈1.7T, Gemini 2.5 Pro ≈1.2T. Adds the quantitative-capacity axis to a concept that previously covered only memory structure.
  • [2026-05-22] STEM: Scaling Transformers with Embedding Modules: Replaces the FFN up-projection with a token-indexed embedding lookup, making the parametric-memory addressing scheme architectural rather than learned. No router, no load-balance loss, table can sit on CPU. Stable training under extreme sparsity (no MoE-style spikes), beats dense baselines while shedding ~1/3 of activated FFN params. Adds the architectural lever on the parametric-memory axis — contrasts with the emergent-geometric structure of Deep sequence models tend to memorize geometrically; it is unclear why.
  • [2026-05-22] Deep sequence models tend to memorize geometrically; it is unclear why: Memorizing a path-star graph and then planning multi-hop paths over it (no CoT, no per-hop supervision) is solved by transformers and Mamba at 10k-node / 10-hop scale because they spontaneously embed nodes geometrically — not because they do associative recall. The bias is shown not to be capacity-driven, not to be reverse-order-learning, and to reduce to a CE-loss low-rank spectral bias already in Node2Vec. Establishes that parametric memory is structurally richer than a flat key-value store.

Open questions

Section titled “Open questions”
  • Is the geometric-memory bias of Deep sequence models tend to memorize geometrically; it is unclear why what actually drives the 2-bits-per-parameter capacity that Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity measures? The capacity ceiling comes from allenzhu2025’s synthetic-fact regime; the geometric-emergence result suggests real corpora may be packed more densely because globally-coherent embedding geometries can store relational structure for free. No filed paper has measured IKP-style accuracy against models trained with frozen vs unfrozen embeddings.
  • Does STEM-style explicit token-indexed embedding storage beat the implicit-geometric-memory baseline at iso-parameter? STEM’s headline is FLOPs/communication efficiency at parity, not knowledge capacity per parameter. If the embedding-table parameters genuinely store ~2 bits/param of factual knowledge while the gate/down stay procedural, the IKP-predicted size of a STEM model would be closer to total params than for a dense model. Not yet tested.
  • The hallucination-similarity fingerprint from Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity measures which wrong answers two models share. If parametric memory is geometric, shared wrong answers should reflect shared embedding geometry — which would give a sharper test for distillation lineage than the existing rare-fact-Jaccard signal. Open question: does the geometric-memory framing predict the distillation-vs-retrain signal IKP empirically observes?
  • Heavily safety-tuned models (Llama 4 Scout 109B predicted as 18B) suggest refusal training can mask parametric memory at the output layer. Does the memory still exist in the weights, recoverable via white-box probes, or has it been overwritten? This connects parametric-memory to unlearning / knowledge-editing.
  • The Densing-Law-falsification result implies that all recent capability gains on standard benchmarks are coming from procedural improvements (better RL recipes, better tool use, better CoT), not from added factual capacity. For products that rely on long-tail factual recall (search, retrieval-light agents, citation), this means parameter count remains the only knob that moves the metric. Open whether this conclusion survives the next generation of post-training tricks that may transfer facts into procedure.
  • In-state vs. in-weights parametric memory. Raven’s slot routing makes memory allocation a runtime decision in the hidden state; STEM makes it an architectural-but-static decision in the FFN; the geometric-memory work shows memory is emergent in weights. Open question: at what context length / recall density does the runtime in-state axis start to dominate the in-weights axis as a capacity multiplier — and does the IKP-style scaling law transfer to recurrent-state slot capacity?
  • How does parametric memory connect to the “Dynamics” axis of agent memory? Memory in the Age of AI Agents names formation (continual training, distillation), evolution (model editing, unlearning), and retrieval (forward-pass activation of stored facts) as first-class operations. None of the filed parametric-memory papers measure all three together — IKP measures retrieval, STEM measures formation, the geometric work measures both implicitly. Open whether a single recipe can move all three simultaneously without interference.

Papers

Section titled “Papers”

2026-04

Section titled “2026-04”

2026-01

Section titled “2026-01”

2025-12

Section titled “2025-12”

2025-11

Section titled “2025-11”

2025-10

Section titled “2025-10”

2025-06

Section titled “2025-06”

2025-05

Section titled “2025-05”