Text-to-LoRA: Instant Transformer Adaption

TL;DR

Text-to-LoRA (T2L) is a Sakana-AI hypernetwork that takes a natural-language description of a task and outputs a LoRA adapter for a target LLM in a single forward pass — no per-task data, no fine-tuning. Trained on a suite of 9 pre-trained task LoRAs (GSM8K, ARC, etc.) via two recipes — reconstruction of the source LoRAs and end-to-end SFT through the adapted LLM — T2L’s generated adapters match the performance of the original task-specific LoRAs they were distilled from. It can also compress hundreds of LoRA instances into one hypernetwork and zero-shot generalize to unseen tasks specified only by text. The compute story: at inference T2L is reported as ~4× cheaper in FLOPs than 3-shot in-context learning on the first instance, with the gap widening as more queries amortize the one-time adapter generation.

Key claims

A hypernetwork conditioned on a natural-language task description can emit task-specific LoRA adapter weights for a target LLM in a single forward pass [Abstract].
After training on 9 pre-trained LoRA adapters (GSM8K, ARC, etc.), reconstructed LoRAs from T2L match the performance of the corresponding task-specific source adapters on their test sets [Abstract].
T2L can compress hundreds of LoRA instances into one hypernetwork and zero-shot generalize to entirely unseen tasks specified only by text [Abstract].
Two training recipes work: (a) reconstruction — distill source LoRA weights given task descriptions, and (b) SFT — train end-to-end through the LoRA-applied target LLM on task data, with the hypernetwork being the only trained component [external summary; appendix references LoRA reconstruction and SFT regimes].
Architecture has three size variants (L / M / S) that differ in parameter count and output structure; the hypernetwork combines a frozen text-embedding model (gte-large-en-v1.5) with learnable module and layer embeddings so a single network can emit per-layer, per-module LoRA factors [external summary].
Adaptation cost is amortized: T2L’s per-instance FLOPs (text-encoder + hypernetwork + base LLM forward) come out to ~0.86 TFLOPs vs. ~4.18 TFLOPs for the same base LLM with 3-shot ICL, a >4× FLOP reduction on the first query and a much larger reduction as queries amortize the one-time hypernetwork forward [Appendix I.2].
Training pool: 500 SNI (Super-Natural-Instructions) datasets from Lots-of-LoRAs; 479 used for training, the rest held out (task IDs 035, 039, 1557, 202, 304, 362, 614, 701, 706, 710, 726) [Appendix J].

Method

T2L is a hypernetwork whose input is a text embedding of a task description (produced by a frozen embedder, gte-large-en-v1.5) plus learnable module and layer embeddings that index which weight matrix’s LoRA factors the hypernetwork should produce on this call. Output is the $(A, B)$ factor pair for one (layer, module) slot of the target LLM; sweeping the module/layer embeddings produces the full adapter set. Three architectural variants (L/M/S) trade hypernetwork size against the granularity of the output factorization.

Two training procedures are presented and shown to be complementary: (1) Reconstruction: distill a pool of pre-trained per-task LoRAs by minimizing reconstruction loss on the source factors given task descriptions — fast, but inherits whatever pathologies the source LoRAs have. (2) SFT: forward the task data through the target LLM with the T2L-generated LoRA applied, backpropagate the standard SFT loss through the LoRA-application path into the hypernetwork weights only. The base LLM is frozen throughout. The reported recipe trains on 9 anchor LoRAs (GSM8K, ARC, etc.) and a larger pool of 479 SNI tasks. Code: github.com/SakanaAI/text-to-lora.

Results

On the 9 anchor tasks, T2L’s reconstructed LoRAs match the source task-specific LoRAs’ performance on their respective test sets [Abstract]. On held-out SNI tasks (out of the ~500-task pool), T2L zero-shot generalizes — a single hypernetwork compresses hundreds of task adapters and produces usable adapters for entirely unseen task descriptions [Abstract]. The FLOP comparison against 3-shot ICL (computed in Appendix I.2 for the M-size variant on a 4096-hidden, 32-layer LLM) is ~0.86 TFLOPs/instance for T2L vs. ~4.18 TFLOPs/instance for ICL — a >4× reduction on the first query alone, and the ICL cost recurs every instance while the hypernetwork forward pass is a one-time amortized cost.

Why it’s interesting

T2L is the task-description-conditioned sibling of Doc-to-LoRA: Learning to Instantly Internalize Contexts (document-conditioned) — together they argue that a hypernetwork can produce useful LoRA adapters from very different conditioning signals (text → task LoRA; document → context-internalized LoRA). For Luma, two angles. First, T2L offers a cleaner deployment story than “carry hundreds of pre-trained per-task LoRAs and route between them” — compressing the LoRA zoo into one hypernetwork is a serving-side simplification that complements the learned-basis approach in Spanning the Visual Analogy Space with a Weight Basis of LoRAs (LoRWeB), with which T2L disagrees about the stability of hypernetwork-generated LoRAs. Second, the FLOP analysis re-frames per-task adaptation as cheaper than ICL once the hypernetwork is trained — relevant when planning routing strategies in multi-task agents like MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling or On Data Engineering for Scaling LLM Terminal Capabilities where per-task specialization is currently done via prompts.