Skip to content
Bud's Wiki

Parametric memory

TL;DR

Section titled “TL;DR”

How facts and rare-entity knowledge are stored in network weights, and what the structure of that storage looks like — strict associative lookup, emergent geometric/relational embeddings, or explicit token-indexed embedding tables — together with the practical implication that the amount of parametric memory scales log-linearly with parameter count at ~2 bits/parameter. The filed papers cluster around three claims: (i) parametric memory in standard transformers is geometric, not just associative, and emerges from a CE-loss spectral bias (Deep sequence models tend to memorize geometrically; it is unclear why); (ii) one can make the lookup architectural by replacing the FFN up-projection with token-indexed embedding tables and recover most of the capacity benefit of MoE without learned routers (STEM: Scaling Transformers with Embedding Modules); and (iii) because factual recall is incompressible, a black-box accuracy score on rare-entity probes lower-bounds the parameter count to within ~1.5× and shows factual capacity is not compressing year-over-year (Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity). The three together justify treating parameter count as the binding resource for long-tail factual recall — even as procedural benchmarks saturate.

Key claims

Section titled “Key claims”
  • Transformers and Mamba spontaneously develop geometric embeddings over memorized graphs — non-co-occurring entities are positioned to reflect global multi-hop distances — turning L-fold compositional recall into a 1-step spatial query, and this happens at scales (10k nodes, 10-hop paths) where strict associative recall would predict failure (Deep sequence models tend to memorize geometrically; it is unclear why §2.1, Fig. 4, Fig. 6).
  • Geometric memory is not strictly more succinct than associative memory — there are graphs where both have equal parameter cost, yet the model still picks geometric — so the bias is a learning bias of CE loss, not a capacity argument; it reduces to the low-rank spectral bias already visible in shallow Node2Vec (Deep sequence models tend to memorize geometrically; it is unclear why §3.1, §5).
  • Replacing the FFN up-projection with a token-indexed embedding lookup E[token_id] (keeping gate and down dense) makes parametric memory architectural: no learned router, no all-to-all, no load-balance loss, and the giant table can live on CPU and be asynchronously prefetched. STEM beats dense baselines while removing ~1/3 of activated FFN params (STEM: Scaling Transformers with Embedding Modules §3, Abstract).
  • STEM’s learned per-token “up-projection vectors” spread out angularly (low pairwise cosine similarity), which the authors connect to reduced representational interference — i.e. parametric memory works better when the addressing scheme spreads keys (STEM: Scaling Transformers with Embedding Modules §4).
  • Storing facts requires at least ~2 bits per parameter (allenzhu2025 / morris2025memorize), so accuracy on rare-entity probes provides a Shannon-entropy lower bound on parameter count that is intrinsic to the model rather than dependent on hardware/serving assumptions (Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity §3.2).
  • IKP calibration on 89 open-weight models (135M–1.6T, 19 vendors) achieves R² = 0.917 with slope ≈0.147 (each 10× in params adds ~14.7pp accuracy on incompressible probes); LOO-CV places 68.5% of held-out open models within 1.5× of true size (Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity §6.1, §6.2).
  • For MoE models, total parameters predict knowledge capacity (R²=0.79) far better than active parameters (R²=0.51) — facts are stored across all expert weights, not just the routed ones; this is the load-bearing empirical claim for “MoE is parametric memory at total-params scale” (Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity §6.1, Fig. 2).
  • Procedural capability compresses under the Densing Law, but factual capacity does not: across 96 dated open-weight models the time coefficient on IKP accuracy is −0.05pp/month (95% CI [−0.20, +0.10]), rejecting Densing’s +1.92pp/month prediction at p<0.001 (Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity §1, Contributions §3).
  • Parametric memory has an in-state analogue distinct from weight storage: in a recurrent linear-time sequence model, the hidden state StS_t itself can be partitioned into independently-addressable slots, with a learned sparse router deciding which slots persist and which are overwritten per token — making the recall-vs-persistence trade explicit rather than emergent (Raven Part 1 — Memory as a Set of Slots).
  • In the agent-memory literature, “parametric” is one of three Forms (alongside token-level and latent) — the substrate axis on which agent memory is layered (Memory in the Age of AI Agents). The survey explicitly distinguishes the parametric-memory work studied here (continual training / model editing / distillation as memory operations) from RAG (a token-level Form) and from the LLM-internal facts studied in isolation from any agentic loop.

Recent contributions

Section titled “Recent contributions”
  • [2026-08-26] Qwen3.8-Flash-Next — Hybrid Attention with QSA, Gated Residual, N-gram Embedding: Qwen3.8-Flash-Next ships a 20,000,000-entry bigram/trigram N-gram Embedding table at layer 2, adding 51B parameters (vs 125B total / 6B active MoE) explicitly framed by Qwen as an alternative parameter-scaling axis to MoE that is “more amenable to offloading” on memory-constrained accelerators. Third filed production instance of architectural parametric memory alongside STEM’s FFN-embedding-table lever and Over-Tokenized Transformer’s hierarchical input-side n-gram table — and the first to land inside a flagship-tier open-weights model.
  • [2026-08-14] Thea: Towards the Harness of Embodied Agents: Thea’s Scene Graph as Context is an external-symbolic + working + formation-heavy memory in the Memory-in-the-Age-of-AI-Agents taxonomy — a persistent world-state graph the agent queries in place of raw perception, distinct from the latent + working + retrieval-heavy pole of long-context agents and from the external-text + experiential + evolution-heavy pole of Training-Free GRPO’s experience library. Context engineering assigns each input a lifetime tier (resident / refreshed / accumulated), naming the Dynamics-axis discipline explicitly.
  • [2026-08-14] Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling: Over-Encoding places a 10710^7-entry hierarchical nn-gram embedding table at the transformer input as a learned lookup memory — the input-side analog of the FFN-as-Hebbian-store framing in MLPs are Hebbians: Constructing Efficient Fact-Storing MLPs for Transformers. Sparse access pattern (one lookup per input token) means embedding-table size is decoupled from FLOPs, so scaling to 10710^7 entries costs <5% throughput with tensor-parallel sharding. Interacts non-trivially with MoE: OLMoE-7B + OE-12.8M shows the same loss improvement as OLMoE-1.3B + OE-12.8M but reduced downstream gains, suggesting sparse-input capacity partially substitutes for sparse-FFN capacity.
  • [2026-08-11] Full-bandwidth transformer: Full-bandwidth transformer adds an in-forward-pass latent-carrier axis that is neither in-weights (STEM, geometric memory) nor in-recurrent-state (Raven, SDM) nor in-context (agent memory): the top-layer hidden state, GLU-fused into the next step’s input embedding, carries non-verbalized computation across the discrete-token bottleneck for one extra depth budget. Structurally closest to Anthropic’s J-space workspace (A global workspace in language models) as a “shared-write substrate within a forward pass”, but with an explicit cross-step recirculation mechanism rather than an emergent one.
  • [2026-08-10] MEM: Multi-Scale Embodied Memory for Vision Language Action Models: MEM formalizes the two-timescale split for embodied policies: dense video-encoded short-horizon memory (temporal attention every 4th ViT layer, zero new parameters, keeps single-image downstream token count) plus a compressed language-based long-horizon memory that the high-level policy predicts and updates step-by-step, with LLM-generated summarization training data actively removing obsolete detail to reduce train-inference distribution shift.
  • [2026-08-06] BridgeVLA++: A Data-Efficient, Generalizable, and Memory-Augmented Vision-Language-Action Framework for 3D Manipulation: BridgeVLA++ stakes out an in-architecture unified spatio-temporal memory module position for VLA memory — ~9.2% param overhead on top of BridgeVLA v1’s aligned-heatmap 3D pipeline — distinct from PrediMem’s external keyframe bank (RoboMemArena: A Comprehensive and Challenging Robotic Memory Benchmark), from RoboTTT’s memory-into-weights via test-time training (RoboTTT — 8k-timestep robot policy via test-time training (Jim Fan / NVIDIA GEAR)), and from ICWM’s in-context system identification (In-Context World Modeling for Robotic Control). Claims to fuse persistent spatial context and temporal interaction history in one module rather than as separate heads.
  • [2026-07-25] SuperMap: A Spatio-Temporal SLAM System for Visual-Language Navigation: SuperMap (CMU, RSS 2026) is a direct sibling to DAAAM at the external-symbolic + experiential + evolution-heavy pole: a training-free SLAM-backed 4D scene graph with probabilistic geometric-consistency + Bayesian semantic fusion managing persistent open-vocabulary object identities across long-horizon deployments (validated on a 2-hour campus run), serialized as structured text for VLM queries — different memory-management substrate from DAAAM’s VLM-caption + factor-graph recipe but same slot in the Forms × Functions × Dynamics taxonomy.
  • [2026-07-22] MLPs are Hebbians: Constructing Efficient Fact-Storing MLPs for Transformers: Hazy Research (Garcia, Liu, Junkins, Eyuboglu, Rudra, Ré) give the first closed-form Transformer-compatible fact-storing MLP construction that hits the ~2 bits/parameter information-theoretic bound IKP measures empirically — no gradient descent, 10–104× fewer params than prior constructions and 15–63× fewer inside a Transformer block, with modular fact-editing by MLP swap. The Hebbian kernel view (similarity in the MLP’s non-linear feature space) sits between the flat-associative store and the emergent-geometric hypothesis of Deep sequence models tend to memorize geometrically; it is unclear why, and complements STEM: Scaling Transformers with Embedding Modules by keeping the standard MLP shape and instead choosing weights closed-form.
  • [2026-07-16] RoboMemArena: A Comprehensive and Challenging Robotic Memory Benchmark: RoboMemArena / PrediMem instantiates an external-store memory form for embodied agents (keyframe bank + recent-frame buffer read by a dual-system VLA) — distinct from the in-weights (STEM, geometric-memory), in-state (Raven, SDM), and in-context (ICWM) forms already tracked here. The novel operational lever is a training-only predictive-coding auxiliary head: supervising the S2 hidden state to predict next-frame ViT features (α=0.1 latent MSE + cosine, stop-gradient teacher) shapes representations so keyframe-write decisions can be made through the standard LM head without a retrieval module, and is discarded at inference for zero runtime cost — an analog to STEM’s “make the addressing scheme architectural” bet, but applied to the write side of an external memory store rather than the read side of a weight table.
  • [2026-07-09] Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity: SDM (Meta/FAIR) makes the initial state of a sparse-addressed linear-RNN memory bank a learned parameter — the first filed instance of architectural parametric memory inside a recurrent state. Distinct from STEM’s FFN-embedding-table lever (static, outside the sequence dimension) and Raven’s slot routing (routes a small dense state rather than growing a memory bank). Reports gains on common-knowledge and reasoning tasks when the initial state is trained end-to-end.
  • [2026-07-08] A global workspace in language models: Anthropic’s J-space (identified by a Jacobian lens on per-token output influence) is a small, densely-connected residual substructure that functions as a broadcast/workspace: a single edit (“France”→“China”) propagates across four different downstream question-answering tasks (capital/language/continent/currency), consistent with one shared representation being read by many downstream circuits rather than per-task copies. Complements longer-timescale parametric-memory work by describing an in-forward-pass shared-write substrate — the temporal analog of caching in a workspace rather than storing in weights.
  • [2026-05-25] Memory in the Age of AI Agents: Positions “parametric” as one Form among three (token-level / parametric / latent) in a broader agent-memory taxonomy that cross-products it with Functions (factual / experiential / working) and Dynamics (formation / evolution / retrieval). Reframes parametric memory as the substrate the agent-memory layer sits on rather than a free-standing object — and explicitly excludes it from “agent memory proper” when there is no agentic loop above it. The Dynamics axis (especially evolution) is the framing the prior parametric-memory work has been missing — model editing, continual training, and distillation are all “parametric-evolution” operations under this taxonomy.
  • [2026-05-24] Raven Part 1 — Memory as a Set of Slots: Raven (Goomba Lab) introduces the Router State Model (RSM) framework — a single state-update equation unifying SSMs (dense router) and SWA (one-hot cyclic router), with Raven as the learned-sparse-router instantiation. Reframes the recall failures of SSMs as a structural problem (uniform decay perturbs every stored item) rather than a capacity problem, and proposes per-slot persistence as the fix. Adds an in-state, learned-routing lever to the parametric-memory toolkit, distinct from the weight-side levers in STEM (architectural FFN replacement) and IKP (fact-density measurement).
  • [2026-05-24] Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity: Builds a black-box parameter-count estimator by treating factual recall as the lower-bound Shannon-entropy signal of parametric memory. Calibrates a log-linear acc = a + b·log10(N) curve at R²=0.917 over 89 open models, separates compressible (procedural) from incompressible (factual) capability, and shows year-over-year factual-capacity scaling has not compressed even as MMLU/HELM saturate. Headline closed-model estimates: GPT-5.5 ≈9T, Claude Opus 4.7 ≈4T, GPT-5.4 ≈2.2T, Claude Sonnet 4.6 ≈1.7T, Gemini 2.5 Pro ≈1.2T. Adds the quantitative-capacity axis to a concept that previously covered only memory structure.
  • [2026-05-22] STEM: Scaling Transformers with Embedding Modules: Replaces the FFN up-projection with a token-indexed embedding lookup, making the parametric-memory addressing scheme architectural rather than learned. No router, no load-balance loss, table can sit on CPU. Stable training under extreme sparsity (no MoE-style spikes), beats dense baselines while shedding ~1/3 of activated FFN params. Adds the architectural lever on the parametric-memory axis — contrasts with the emergent-geometric structure of Deep sequence models tend to memorize geometrically; it is unclear why.
  • [2026-05-22] Deep sequence models tend to memorize geometrically; it is unclear why: Memorizing a path-star graph and then planning multi-hop paths over it (no CoT, no per-hop supervision) is solved by transformers and Mamba at 10k-node / 10-hop scale because they spontaneously embed nodes geometrically — not because they do associative recall. The bias is shown not to be capacity-driven, not to be reverse-order-learning, and to reduce to a CE-loss low-rank spectral bias already in Node2Vec. Establishes that parametric memory is structurally richer than a flat key-value store.

Open questions

Section titled “Open questions”
  • Is the geometric-memory bias of Deep sequence models tend to memorize geometrically; it is unclear why what actually drives the 2-bits-per-parameter capacity that Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity measures? The capacity ceiling comes from allenzhu2025’s synthetic-fact regime; the geometric-emergence result suggests real corpora may be packed more densely because globally-coherent embedding geometries can store relational structure for free. No filed paper has measured IKP-style accuracy against models trained with frozen vs unfrozen embeddings.
  • Does STEM-style explicit token-indexed embedding storage beat the implicit-geometric-memory baseline at iso-parameter? STEM’s headline is FLOPs/communication efficiency at parity, not knowledge capacity per parameter. If the embedding-table parameters genuinely store ~2 bits/param of factual knowledge while the gate/down stay procedural, the IKP-predicted size of a STEM model would be closer to total params than for a dense model. Not yet tested.
  • The hallucination-similarity fingerprint from Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity measures which wrong answers two models share. If parametric memory is geometric, shared wrong answers should reflect shared embedding geometry — which would give a sharper test for distillation lineage than the existing rare-fact-Jaccard signal. Open question: does the geometric-memory framing predict the distillation-vs-retrain signal IKP empirically observes?
  • Heavily safety-tuned models (Llama 4 Scout 109B predicted as 18B) suggest refusal training can mask parametric memory at the output layer. Does the memory still exist in the weights, recoverable via white-box probes, or has it been overwritten? This connects parametric-memory to unlearning / knowledge-editing.
  • The Densing-Law-falsification result implies that all recent capability gains on standard benchmarks are coming from procedural improvements (better RL recipes, better tool use, better CoT), not from added factual capacity. For products that rely on long-tail factual recall (search, retrieval-light agents, citation), this means parameter count remains the only knob that moves the metric. Open whether this conclusion survives the next generation of post-training tricks that may transfer facts into procedure.
  • In-state vs. in-weights parametric memory. Raven’s slot routing makes memory allocation a runtime decision in the hidden state; STEM makes it an architectural-but-static decision in the FFN; the geometric-memory work shows memory is emergent in weights. Open question: at what context length / recall density does the runtime in-state axis start to dominate the in-weights axis as a capacity multiplier — and does the IKP-style scaling law transfer to recurrent-state slot capacity?
  • How does parametric memory connect to the “Dynamics” axis of agent memory? Memory in the Age of AI Agents names formation (continual training, distillation), evolution (model editing, unlearning), and retrieval (forward-pass activation of stored facts) as first-class operations. None of the filed parametric-memory papers measure all three together — IKP measures retrieval, STEM measures formation, the geometric work measures both implicitly. Open whether a single recipe can move all three simultaneously without interference.

Papers

Section titled “Papers”

2026-04

Section titled “2026-04”

2026-01

Section titled “2026-01”

2025-12

Section titled “2025-12”

2025-11

Section titled “2025-11”

2025-10

Section titled “2025-10”

2025-06

Section titled “2025-06”

2025-05

Section titled “2025-05”