Doc-to-LoRA: Learning to Instantly Internalize Contexts

TL;DR

Doc-to-LoRA (D2L) is a Sakana-AI hypernetwork that meta-learns to approximate context distillation in a single forward pass: given an arbitrary long document, it emits a LoRA adapter for a target LLM, so subsequent queries can be answered without re-feeding the document into the prompt. The architecture uses a Perceiver-style cross-attention backbone with a chunking mechanism that lets it produce higher-rank LoRA adapters for longer inputs without changing the hypernetwork’s output shape, enabling generalization to contexts well beyond its training length. On a needle-in-a-haystack benchmark, D2L achieves near-perfect zero-shot accuracy at sequence lengths exceeding the target LLM’s native context window by more than 5×, and on real-world QA it outperforms standard context distillation at limited compute while reducing peak memory and update latency. Conceptually, this relocates long-context memory from the KV cache into adapter weights — a different shape of “context-as-weights” than test-time training or KV-cache compression.

Key claims

A hypernetwork can meta-learn the context-distillation map (long prompt → weight update) and produce a usable LoRA adapter for an unseen document in a single forward pass, eliminating the per-prompt optimization cost of conventional context distillation [Abstract, §1].
The hypernetwork is built on a Perceiver backbone so it can ingest variable-length contexts and emit a fixed-shape latent, then unfolds that latent into LoRA factors; the variable input → fixed output decoupling is what enables length generalization [§1, method body].
A chunking mechanism lets the hypernetwork produce higher-rank LoRA adapters for longer documents (effectively concatenating per-chunk adapter slices) without modifying the hypernetwork’s output shape, which the authors identify as the crucial trick for documents that exceed the target LLM’s own context window [§1].
On a synthetic Needle-in-a-Haystack task, Doc-to-LoRA achieves near-perfect zero-shot accuracy on contexts beyond 5× the target LLM’s native context window, despite being trained only on sequences up to the native window length [Abstract, §1].
On real-world QA datasets, Doc-to-LoRA outperforms standard per-prompt context distillation under a fixed query budget while reducing peak memory consumption and update latency [Abstract].
At inference time, the LLM consumes only the LoRA-modified weights — no KV cache for the original context — so memory and latency are decoupled from document length once the adapter has been generated [Abstract].

Method

Doc-to-LoRA is a hypernetwork (Ha et al. 2016) trained to approximate the result of context distillation: given a document $D$ , output a LoRA adapter $\Delta W = BA$ such that the target LLM with weights $W + \Delta W$ behaves as if it had ingested $D$ via prompt. Training is meta-learning: the outer loop optimizes the post-adapter loss of the target LLM on queries about $D$ , sampled across many documents. The hypernetwork architecture is a Perceiver — variable-length token input cross-attended into a fixed-size latent array — chosen specifically so a single hypernetwork can handle documents of arbitrary length without padding/truncation. For documents longer than what one Perceiver call can comfortably absorb, D2L chunks the document, runs the hypernetwork on each chunk to obtain per-chunk LoRA factor blocks, and concatenates the chunk outputs along the rank axis to form a single higher-rank LoRA adapter — i.e. longer documents get higher-rank adapters at no change to the per-chunk hypernetwork shape.

At inference there are two passes: (1) one forward pass through the hypernetwork to materialize the LoRA, and (2) standard inference through the target LLM with the adapter applied. The KV cache for the document itself is never built. Code at github.com/SakanaAI/doc-to-lora.

Results

The headline benchmark is a synthetic Needle-in-a-Haystack (NIAH) task — insert a fact at a random position in a long distractor passage, ask about it later. Trained on sequences up to the target LLM’s native context window, D2L generalizes to sequences >5× that length with near-perfect zero-shot accuracy [Abstract, §1]. The training-length generalization is attributed to the Perceiver + chunking combination — the hypernetwork only ever sees fixed-shape chunks, and longer documents just means more chunks → higher-rank adapter.

On real-world QA benchmarks under a limited query budget per document, D2L outperforms a standard context-distillation baseline (per-prompt fine-tuning) on accuracy, while consuming dramatically less peak memory and finishing the “update” step in a single forward pass instead of an optimization loop [Abstract, training+downstream figure caption]. Headline framing from the announcement: sub-second adaptation latency, including the visual-information-into-text-only-LLM transfer demonstration where a hypernetwork conditioned on VLM features produces a LoRA that lets a text-only LLM classify images.

Why it’s interesting

Doc-to-LoRA is the cleanest current example of a third axis for long context that is neither “build a longer-context architecture” nor “compress the KV cache” — it amortizes per-document context distillation into a hypernetwork’s forward pass. That makes it directly comparable to End-to-End Test-Time Training for Long Context‘s TTT-E2E (which also moves context into weights but via gradient steps at test time) and to Inference-Time Hyper-Scaling with KV Cache Compression (which keeps the cache but compresses it). The team should care because the design point — fixed-shape Perceiver + chunk-rank scaling — is a recipe that maps cleanly onto other “ingest long thing once, query many times” workloads: long video conditioning, long planning traces, persona/style internalization. The LoRWeB observation in Spanning the Visual Analogy Space with a Weight Basis of LoRAs that learned bases of LoRAs are a more stable replacement for per-task hypernetworks is the obvious tension to track — D2L revives the hypernetwork-LoRA approach with a specific architectural recipe (Perceiver + chunking + meta-RL-style outer loop) that LoRWeB’s framing argued against.