Generalization Dynamics of LM Pre-training

TL;DR

The paper builds a toy eval suite that probes whether an LM at a given checkpoint is doing “parrot-like” (memorization / shallow pattern-matching / System-1) computations or “intelligence-like” (in-context learning / System-2 / true multi-hop) computations across matched task pairs, and runs it densely across pretraining. The main empirical finding is mode-hopping: throughout pretraining an LM suddenly and repeatedly flips between the two computation modes on the same task — latching onto memorized patterns, System-1 shortcuts, or truthiness-over-truth, then just as suddenly reverting to generalization. Mode-hops are locally stable (checkpoint averaging doesn’t fix them) and are not explained by standard optimization dynamics, which the authors reframe as a capacity-allocation problem: shallow circuits learned early compete with generalizable circuits for bounded capacity, and each pretraining data window can tip the balance. Two practical payoffs follow — selecting an intermediate checkpoint that generalizes reasoning and alignment better than the final pre- or mid-training checkpoint, and selecting pretraining data windows that stabilize generalization dynamics.

Key claims

Throughout pretraining, LMs frequently and suddenly hop between parrot-like and intelligence-like computations on the same task; the mental model of stable maturation from memorization to generalization is empirically wrong [Abstract].
Mode-hopping manifests concretely as: latching onto memorized or in-context patterns instead of doing in-context learning; picking System-1 over System-2; picking what “sounds true” over what is true; failing at multi-hop persona QA, out-of-context reasoning, and emergent misalignment — then reverting [Abstract].
Mode-hops are locally stable: they cannot be smoothed out by standard checkpoint averaging, ruling out a pure optimization-noise explanation [Abstract].
The proposed mechanistic account is a capacity-allocation problem — generalizable circuits must compete with shallow circuits learned early in training for bounded model capacity, and the data present in a given pretraining window decides which wins [Abstract].
Selecting intermediate pretraining checkpoints via the toy eval suite yields models with stronger reasoning and alignment generalization than either the final pre-training or the final mid-training checkpoint [Abstract, application (i)].
Selecting pretraining data using the same lens controls and stabilizes generalization dynamics, i.e. reduces the frequency and severity of mode-hops [Abstract, application (ii)].

Method

The authors construct a toy eval suite deliberately built to distinguish two computation modes per task — a parrot-like mode (memorization, in-context pattern match, System-1, “sounds true”) and an intelligence-like mode (in-context learning, System-2, multi-hop composition, out-of-context reasoning, truthfulness) — across a range of task families that each admit both a shortcut and a generalizing solution. The suite is then applied densely across pretraining, so that the trajectory of each checkpoint’s mode-mixture can be plotted rather than only its endpoints. Checkpoint averaging is used as a diagnostic to test whether mode-hops are optimization-noise artifacts (they aren’t). The capacity-allocation framing is developed as an interpretive story rather than as a formal model. On the applications side, the suite is used as a selection criterion: (i) pick the intermediate checkpoint that scores best on the generalization sub-scales; (ii) filter/reweight pretraining data windows to favor those that push the model toward the intelligence-like mode.

Results

Qualitatively: on the paper’s suite, LMs exhibit repeated, sudden transitions between parrot-mode and intelligence-mode on the same task across pretraining, without the transitions being predictable from loss or standard scaling-law extrapolation [Abstract].
Checkpoint averaging does not remove mode-hops, evidence that they are not high-frequency optimization noise [Abstract].
The intermediate-checkpoint selection procedure produces models with strictly better reasoning and alignment generalization than the final pre- or mid-training checkpoint [Abstract, application (i)] — implying the standard practice of shipping the final checkpoint leaves generalization on the table.
The data-selection procedure demonstrably reduces mode-hop severity, i.e. the same suite serves as both diagnostic and training-data signal [Abstract, application (ii)].
(Quantitative deltas, model families, and sizes are not surfaced in the abstract text available at filing; the specific FLOP / parameter / benchmark numbers should be added on re-read from the full PDF.)

Why it’s interesting

The paper reframes a familiar informal observation — “training runs jump around before they settle” — as a reproducible eval suite and then treats the resulting mode-hopping signal as a first-class artifact to select on. This puts it in direct dialogue with Fractal basins trap latent reasoning, which found that reasoning traces converge with a fractal basin structure that strengthens with task difficulty and that a controlled loop-transformer run exhibited fractal-basin emergence at the same mid-training bifurcation as generalization — the two papers together suggest that the “sudden bifurcation” story is real at both the pretraining-dynamics scale (this paper) and the inference-dynamics scale (fractal basins). It also complements the emergent-misalignment thread (Training large language models on narrow tasks can lead to broad misalignment, Natural Emergent Misalignment from Reward Hacking in Production RL) by locating a pre-post-training substrate for the same phenomenon — if generalization is unstable during pretraining, then post-training misalignment may be riding a wave that started earlier. Finally, the “select an intermediate checkpoint” application challenges the wiki’s default assumption (baked into most scaling-law and stability work — Scaling Laws for Neural Language Models, How to Set the Learning Rate for Large-Scale Pre-training?) that the final checkpoint on the compute frontier is the right thing to ship.