Skip to content
Bud's Wiki

Long context as weights

TL;DR

Section titled “TL;DR”

A cluster of recipes that relocate long-context memory out of the KV cache and into network weights — either by gradient updates at test time over the context tokens themselves, by a hypernetwork that emits a document-specific LoRA in a single forward pass, or by an RL-trained policy that generates synthetic finetuning data from the context and applies LoRA SFT. Distinct from KV-cache compression (which keeps the cache and shrinks it) and from linear-attention recurrent architectures (which use a fixed-size hidden state). The three filed instances disagree about whether the weight-update needs to be SGD on the raw context (TTT-E2E), an amortized hypernetwork forward pass (D2L), or an RL-trained generative reformulation of the context into synthetic data (SEAL).

Key claims

Section titled “Key claims”
  • Context information can be pushed into weights at test time via plain next-token-prediction SGD over a meta-learned initialization: TTT-E2E matches full-attention loss-vs-context-length scaling at 3B / 164B tokens on a standard Transformer with sliding-window attention, while keeping RNN-style constant inference latency (2.7× wall-clock over full attention at 128K) (End-to-End Test-Time Training for Long Context Abstract, Fig. 1).
  • The end-to-end meta-learning of the initialization through the inner-loop TTT updates is load-bearing: dynamic-evaluation-style methods that only train the init conventionally and run TTT only at test time do not match this scaling (End-to-End Test-Time Training for Long Context §1, Fig. 1 caption).
  • A hypernetwork conditioned on a long document can emit a document-internalizing LoRA in a single forward pass — no test-time gradient steps — achieving near-perfect zero-shot accuracy on needle-in-a-haystack at sequence lengths >5× the target LLM’s native context window, and removing the KV cache for that document entirely (Doc-to-LoRA: Learning to Instantly Internalize Contexts Abstract, §1).
  • The weight-update can also be parameterized by the LLM’s own generation: SEAL trains a self-edit policy via ReST to generate synthetic implications from the context, applies LoRA SFT, and beats GPT-4.1-generated synthetic data on no-context SQuAD QA (47.0% vs 46.3%) despite the policy being only Qwen2.5-7B (Self-Adapting Language Models Table 2, §4.2).
  • All three weight-update recipes target the same operating point — long-context information available without paying the inference cost of carrying the tokens in context — but differ in the granularity of the update parameter: full backbone (End-to-End Test-Time Training for Long Context, TTT-E2E), document-specific LoRA (Doc-to-LoRA: Learning to Instantly Internalize Contexts, D2L), self-generated implication corpus + LoRA (Self-Adapting Language Models, SEAL).
  • The “context-as-weights” axis is reported separately stable from KV-cache compression: TTT-E2E argues the long-context degradation of linear-attention recurrent architectures (Mamba 2, Gated DeltaNet) is not intrinsic to “weights as memory” but to those particular recurrent updates (End-to-End Test-Time Training for Long Context §1, Fig. 1 left).
  • Sequential context-as-weights edits exhibit catastrophic forgetting: in SEAL, accuracy on earlier passages decays as new edits accumulate, with no retention objective (Self-Adapting Language Models §5, Fig. 6). D2L and TTT-E2E do not report multi-document forgetting curves; whether the hypernetwork-amortized and end-to-end-meta-learned variants suffer the same forgetting is open.
  • Cost profile differs sharply across the cluster: D2L is one hypernetwork forward pass (cheap, but requires training the hypernetwork up front); TTT-E2E is multiple gradient steps on the full backbone per context (expensive at test time, simple to deploy); SEAL is multiple LoRA SFT runs inside the RL outer loop at training time (30–45s per self-edit evaluation), then one LoRA SFT run at test time (Self-Adapting Language Models §5, §B.5).
  • Adjacent substrate — vision tokens, not weights: Long-context memory can alternatively be relocated into a compressed visual representation by rendering text as an image and encoding it through a vision encoder with a high token-reduction bottleneck; DeepSeek-OCR reaches ~97% decoding precision at <10× text-to-vision-token compression and ~60% at 20× on the Fox benchmark, and the authors frame the resulting compression curve as a quantitative starting point for studying LLM forgetting mechanisms (aged context could be re-rendered at progressively lower image resolution) (DeepSeek-OCR: Contexts Optical Compression §4.1, Table 2). This is a sibling axis to the three weight-update recipes — same operating goal, different substrate — and is currently the only filed datapoint on the optical-compression substrate. Karpathy’s commentary (Karpathy: pixels may be better LLM inputs than text tokens (DeepSeek-OCR springboard)) generalizes this from “long-context memory substrate” to “universal LLM input interface”: pixels compress more densely, carry layout/styling/embedded images natively, admit bidirectional input attention, and delete the tokenizer.

Recent contributions

Section titled “Recent contributions”
  • [2026-07-17] WAM-TTT: Steering World-Action Models by Watching Human Play at Test Time: WAM-TTT (Feng, Lyu, Wang et al.) is the first filed paper porting the SGD-on-backbone pole of this cluster (End-to-End Test-Time Training for Long Context) to robot foundation models — the RoboTTT — 8k-timestep robot policy via test-time training (Jim Fan / NVIDIA GEAR) RoboTTT recipe made concrete. Human-demonstration videos are absorbed into a lightweight adaptive key–value memory inside a frozen World-Action Model by self-supervised video prediction at test time; the backbone stays frozen and the memory is where the “context in weights” lives. The novel piece relative to language-domain TTT is a paired-human-robot meta-training stage with a KV memory reconstruction objective — an alignment signal that ensures the human-video TTT updates land at features that are useful for robot control, answering the same “how does the update stay task-useful” question that SEAL (Self-Adapting Language Models) handles with an RL policy. Adds a robotics-domain datapoint to the update-granularity space the page tracks: fine-grained KV memory (per-task/preference) sitting between full-backbone SGD and hypernetwork-emitted LoRA.
  • [2026-07-15] RoboTTT — 8k-timestep robot policy via test-time training (Jim Fan / NVIDIA GEAR): RoboTTT (Jim Fan / NVIDIA GEAR launch tweet) ports the TTT-E2E recipe from language to robot control: a small TTT “core” takes one gradient step per incoming sensor reading, compressing 8,000 timesteps (~5 min) of history into its fixed-size weights with constant per-step inference cost — the first robotics instance of the “memory into weights” cluster. Reported Context Scaling Curve is monotonic from 128 → 8K timesteps with 8K-context pretraining beating 1K by 62% closed-loop, and the same mechanism enables one-shot in-context learning from a single human demonstration video plus mid-episode error recovery. Numbers are tweet-only; deep-dive thread / report deferred and catastrophic-forgetting curve (the failure mode SEAL reports for the recipe family) is not addressed.
  • [2026-07-09] Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity: SDM’s learned-initial-state variant sits adjacent to this cluster: the memory bank is “context in weights” but addressed sparsely per token rather than emitted as a full LoRA (D2L) or updated by full-backbone SGD (TTT-E2E). Distinct trade-off in the update-granularity space this page tracks — per-slot sparse-addressed weight storage baked in at pretrain time.
  • [2026-05-25] Karpathy: pixels may be better LLM inputs than text tokens (DeepSeek-OCR springboard): Karpathy’s pixels-as-LLM-input tweet — uses DeepSeek-OCR as a springboard for a broader claim that pixels (not text tokens) should be the universal input substrate for LLMs. Four arguments: (1) compression — the DeepSeek-OCR-style ratio applies to any input, not just OCR; (2) generality — formatting, layout, and embedded images flow through the same channel; (3) bidirectional input attention by default; (4) no tokenizer, no Unicode baggage. Asymmetric across the I/O boundary — argument is specifically about inputs; text is fine for outputs. Does not move this cluster’s empirical bar but is the strongest non-paper articulation that the optical substrate isn’t a curiosity adjacent to the weight-update axis but a candidate replacement for the text-token interface entirely.
  • [2026-05-25] DeepSeek-OCR: Contexts Optical Compression: DeepSeek-OCR — the original “Contexts Optical Compression” paper, predecessor of DeepSeek-OCR 2 already filed at DeepSeek-OCR 2: Visual Causal Flow. Adds a sibling substrate to the page’s existing three (SGD-on-backbone, hypernetwork-LoRA, RL-distilled-LoRA): compress text into vision tokens via a SAM+conv+CLIP encoder bottleneck. The Fox-benchmark compression curve (~97% at ≤10×, ~60% at 20×) is the first quantitative measurement of how lossy this substrate is, and the authors explicitly propose it as a forgetting-mechanism primitive — aged context re-rendered at progressively lower image resolution. Adjacent to the page’s weight-update axis rather than replacing it; the obvious head-to-head (D2L’s hypernetwork-LoRA vs DeepSeek-OCR’s vision-token bottleneck at matched token-saved budget) has not been filed.
  • [2026-05-24] Self-Adapting Language Models: SEAL — an RL-trained policy generates “self-edits” (natural-language directives) that produce synthetic finetuning data from the context, applied as LoRA SFT updates. ReST is used instead of GRPO/PPO (explicitly reported as unstable for this setup). On SQuAD no-context QA the self-generated implications beat GPT-4.1’s; on a curated ARC subset the policy hits 72.5% vs 20% for unrolled self-edits. Adds the “RL-trained reformulation of context into synthetic data” lever to a cluster previously containing only the SGD (TTT-E2E) and hypernetwork (D2L) levers. Also surfaces catastrophic forgetting under sequential edits as the first quantitative failure mode for this concept.
  • [2026-05-23] End-to-End Test-Time Training for Long Context: TTT-E2E — long-context language modeling cast as continual learning on a standard sliding-window Transformer, with the outer-loop training objective differentiating through the inner-loop TTT next-token updates. Matches full-attention loss-vs-context-length scaling at 3B / 164B tokens; constant inference latency, 2.7× over full attention at 128K. Reframes the long-context fight as architecture-vs-continual-learning rather than as a competition between attention variants.
  • [2026-05-23] Doc-to-LoRA: Learning to Instantly Internalize Contexts: Doc-to-LoRA — a Perceiver-backbone hypernetwork with chunking emits a document-internalizing LoRA in one forward pass, with no test-time gradient steps. Achieves near-perfect zero-shot accuracy on needle-in-a-haystack at >5× the target LLM’s native context window, removing the document’s KV cache entirely. The amortized-hypernetwork pole of the cluster.

Open questions

Section titled “Open questions”
  • Update granularity: at what context length / recall density does the cost-per-token-saved curve favor each pole? D2L (one hypernetwork pass) wins on inference compute but needs upfront training; TTT-E2E (SGD on backbone) is simple but expensive per query; SEAL (RL-trained synthetic-data + LoRA SFT) sits in between. No filed paper benchmarks the three under a matched compute budget on the same task.
  • Hypernetwork vs SGD vs generative-then-SGD at iso-parameter: how does the quality of the weight update compare across the three parameterizations? SEAL’s headline beats GPT-4.1-generated synthetic data, but the comparison to a hypernetwork-emitted LoRA or to direct backbone SGD on the same passage has not been done.
  • Catastrophic forgetting: only SEAL reports a multi-edit retention curve (Self-Adapting Language Models §5). Whether the meta-learned init in TTT-E2E and the chunked Perceiver in D2L exhibit the same decay, and whether the RL outer loop could be reward-shaped to penalize forgetting (the SEAL authors’ suggested direction), are open.
  • Why does ReST work where GRPO and PPO don’t (Self-Adapting Language Models §3.1)? The stop-gradient on the reward is the textbook explanation, but on-policy methods routinely tolerate non-differentiable rewards. The instability may instead be specific to the inner-loop-dependent reward (where the reward depends on the model’s own current parameters via SFT). This is the most direct filed counterexample to GRPO-as-default in Reasoning RL.
  • Compounding adapters: can D2L (document) and SEAL (RL-distilled implications) emit LoRAs that compose? Across-paper compounding hasn’t been tested.
  • Unlabeled corpora: SEAL is explicitly bottlenecked by needing a paired downstream task per context for reward computation (Self-Adapting Language Models §5). D2L sidesteps this by training the hypernetwork supervised on synthetic QA; TTT-E2E uses next-token prediction itself as the inner loss. Whether SEAL’s model-generated-QA workaround actually scales is the obvious next experiment.
  • Boundary vs Parametric memory: at what point does “long context as weights” become indistinguishable from continual pretraining on a curriculum? The SEAL CPT result (n=200 passages, 43.8% vs 32.2% passage-only) is the closest filed datapoint — relative gains hold, but absolute accuracy drops with cross-passage gradient interference. Whether the failure mode is shared across the three recipes is open.
  • Optical compression vs weight-update at matched substrate cost: DeepSeek-OCR’s ~10× compression at ~97% precision (DeepSeek-OCR: Contexts Optical Compression Table 2) sets a quantitative bar that weight-update recipes should be benchmarked against — how many tokens of cache (or LoRA params) does D2L need to recover the same passage at the same fidelity, and at what test-time cost? The two substrates have never been compared head-to-head on a shared benchmark.
  • Is DeepSeek-OCR’s forgetting-mechanism framing testable as written? The paper proposes re-rendering aged context at progressively lower image resolution as a hypothesis (DeepSeek-OCR: Contexts Optical Compression §1, §4.1) but does not run a multi-tier eval; an LLM that ingests Tiny-rendered older context + Gundam-rendered recent context would isolate whether the compression curve gives a usable retention/cost tradeoff in practice.
  • Karpathy’s pixel-as-input thesis vs the wiki’s current encoder factoring (Karpathy: pixels may be better LLM inputs than text tokens (DeepSeek-OCR springboard)): if pixels are the universal input substrate, the standard VLM design (separate text encoder + image encoder feeding a shared transformer) is the wrong factoring. The argument intersects Unified Multimodal Models from a different direction than the unified-tokenizer line of work — the claim isn’t about better tokens, it’s about deleting the text-token branch. No filed paper has trained an LLM input-only on rendered text and measured the loss-vs-compute curve against a text-tokenizer-equivalent baseline.

Papers

Section titled “Papers”

2026-02

Section titled “2026-02”

2026-01

Section titled “2026-01”

2025-12

Section titled “2025-12”

2025-11

Section titled “2025-11”

2025-10

Section titled “2025-10”

2025-06

Section titled “2025-06”