Self-Adapting Language Models

TL;DR

SEAL trains an LLM via reinforcement learning to generate its own “self-edits” — natural-language directives that produce synthetic finetuning data and (optionally) optimization hyperparameters, applied as LoRA SFT updates to the model itself. The outer loop is ReST (rejection sampling + SFT) with binary downstream-task reward; the inner loop is a real gradient update on the model whose performance defines the reward. On SQuAD knowledge incorporation the self-generated implications beat GPT-4.1-generated synthetic data (47.0% vs 46.3%) despite the policy being Qwen2.5-7B, and on a filtered ARC-AGI subset SEAL hits 72.5% success vs 20% for unrolled self-edits without RL and 0% for plain ICL.

Key claims

Self-edits — model-generated natural-language directives specifying synthetic training data (and optionally LR / epochs / loss masking) — can be applied as LoRA SFT updates to make persistent parameter changes; the same primitive instantiates both knowledge incorporation (passage → implications) and ARC-style few-shot adaptation (demos → tool-call augmentation spec) [§3.1, §3.2].
ReST (rejection sampling + SFT, an EM-flavored on-policy method) trains the self-edit policy stably where GRPO and PPO did not: the authors report instability with both GRPO and PPO and adopt ReST as a “filtered behavior cloning” approximation of the binary-reward RL objective [§3.1, Eq. 2–4].
On SQuAD knowledge incorporation (Qwen2.5-7B, no-passage-in-context QA), 2 rounds of ReST take accuracy from 33.5% (passage-only finetune) to 47.0%, beating GPT-4.1-generated synthetic data (46.3%) and the un-RL’d base policy’s synthetic data (39.7%) [Table 2, §4.2].
The knowledge-incorporation policy generalizes from single-passage training to continued-pretraining (n=200 passages, 974 questions): SEAL reaches 43.8% CPT accuracy vs 39.4% for GPT-4.1 synthetic data and 32.2% for passage-only finetune [Table 2, §4.2].
On a curated ARC-AGI subset (11 train tasks, 8 eval tasks) using Llama-3.2-1B-Instruct, SEAL hits 72.5% success vs 20% for TTT+self-edit without RL and 0% for in-context learning; the upper bound (Oracle TTT with hand-picked augmentation config) is 100% [Table 1, §4.1].
The policy collapses quickly: 2 outer RL iterations suffice to overtake GPT-4.1’s synthetic-data baseline, with diminishing returns afterwards — qualitative examples suggest the policy converges to distilling passages into “easily learnable atomic facts” [Fig. 4, Fig. 5, §4.2].
Sequential self-edits exhibit catastrophic forgetting — accuracy on earlier passages decays as new edits accumulate — but the model does not collapse entirely; the authors explicitly do not optimize for retention [Fig. 6, §5].
TTT-style reward loops are 30–45s per self-edit evaluation because each reward requires a full LoRA finetune + eval, making this 1–2 orders of magnitude more expensive per RL step than preference-pair or regex-verified rewards [§5, §B.5].
The “context-dependent evaluation” limitation is structural: SEAL requires a paired downstream task τ per context c to compute reward, which prevents scaling to unlabeled corpora; the authors suggest model-generated QA as the future fix [§5].

Method

SEAL is a two-nested-loop algorithm. The outer loop is an RL update on a self-edit generation policy: given a context c (a passage to memorize, or a few-shot demo set), the model samples a self-edit SE ~ π_θ(·|c). SE is a token sequence that the system parses into (i) synthetic training data (implications, QA pairs, augmented demos) and (ii) optionally training-config tool calls (LR, epochs, loss-mask). The inner loop applies SE: it runs LoRA-based SFT on the synthetic data to produce θ’ = SFT(θ, SE), then evaluates θ’ on the held-out task τ associated with c. The downstream-task reward r ∈ {0, 1} is fed back to the outer loop, which uses ReST — sample multiple self-edits per context, keep only the ones with r=1, run SFT on those — to update the policy. The authors derive ReST as an on-policy EM approximation of the binary-reward gradient, with a stop-gradient on the reward term (which itself depends on θ through the inner loop and is not differentiable through SFT).

For knowledge incorporation the self-edit prompt is “List several implications derived from the content,” and the recipe builds on deductive-closure training (Akyürek et al., 2024) but trains the implication-generator with RL rather than using a fixed prompt. For ARC the self-edit names tools from a fixed library — rotations, flips, transposes, resizes, chained transforms, plus LR / epochs / loss-token-mask — that get applied to the few-shot demos as augmentations before LoRA SFT. The trained policy is Qwen2.5-7B for knowledge incorporation and Llama-3.2-1B-Instruct for ARC.

Results

Knowledge incorporation, single passage (n=1): Base model 32.7% → Passage-only finetune 33.5% → Passage + base-policy synthetic implications 39.7% → Passage + GPT-4.1 synthetic implications 46.3% → SEAL 47.0% on no-context SQuAD QA [Table 2].
Knowledge incorporation, continued pretraining (n=200): Base 32.7 → Passage-only 32.2 → Base-policy synthetic 41.0 → GPT-4.1 synthetic 39.4 → SEAL 43.8 (a 4.4-point gap over GPT-4.1; the absolute scores drop relative to single-passage, attributed to gradient interference across passages) [Table 2].
Few-shot ARC (Llama-3.2-1B): ICL 0% / TTT+SE without RL 20% / SEAL 72.5% / Oracle TTT 100% — i.e., RL closes 73% of the gap to the human-tuned augmentation oracle, from a starting point where ICL is at chance [Table 1].
Convergence: SEAL overtakes GPT-4.1 synthetic data after 2 outer RL iterations; subsequent iterations yield diminishing returns [Fig. 4].

Why it’s interesting

This is the most explicit instantiation so far of “context → weights” as a learned policy: the same family as Doc-to-LoRA: Learning to Instantly Internalize Contexts (D2L: hypernetwork emits a document-specific LoRA in one forward pass) and End-to-End Test-Time Training for Long Context (TTT-E2E: gradient steps at test time to internalize long contexts), but it sidesteps both by using the LLM’s own generation as the parameterization of the update, and trains it with RL rather than supervised hypernetwork training. The synthetic-data angle complements Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing‘s closed-loop multi-agent data generation: there the agent loop is fixed and GRPO trains the consumer; here a single model is both generator and consumer, with ReST gating the data. The choice of ReST over GRPO is itself noteworthy given Reasoning RL‘s heavy GRPO bias — the explicit “tried GRPO and PPO, they were unstable” report is one of the few filed datapoints against the GRPO default for non-standard RL setups. The catastrophic-forgetting story in §5 is also the cleanest filed example of why parametric self-update remains hard even when the update mechanism works in isolation.