Physics of Language Models: Part 4.2, Canon Layers at Scale where Synthetic Pretraining Resonates in Reality

TL;DR

Part 4.2 is the real-world counterpart to Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers: it asks whether Canon layers and the architectural rankings discovered in the synthetic playground actually transfer to controlled, repeatable real-life pretraining at 1.3B / 3B / 8B parameters on 1–2T tokens of the Nemo corpus. Sixteen controlled Llama vs. LlamaCanon runs (three model sizes × multiple LRs) plus a 48-checkpoint linear-attention track (GLA5, GDN2, Mamba2, each with and without Canon) form a single grid where only the architectural delta varies. The headline: Llama+Canon consistently beats Llama and many open baselines on MMLU and retrieval; GLA5+Canon ≈ GDN2+Canon, both beat Mamba2+Canon, and the differences between linear architectures collapse once Canon is added — exactly mirroring the synthetic predictions of Part 4.1. Linear models still lag Transformers on retrieval-heavy tasks and 2-hop reasoning never emerges, even at 8B / 1T tokens.

Key claims

Real-life pretraining at 1.3B / 100B tokens is too noisy to rank architectures; Part 4.2 fixes this by going to 1.3B → 3B → 8B parameters on 1–2T tokens with super-aligned training pipelines (same Nemo data, same tokenizer, multiple LR points per architecture) so the only axis of variation is architecture [Part 4.2 §Setup; tweet (13/N)].
16 controlled runs (1.3B / 3B / 8B × multiple LRs) show Llama+Canon consistently outperforms Llama and many open baselines including SmolLM / Llama3-base / OLMo at matched compute — particularly on MMLU, retrieval, and length generalization [Part 4.2 §Llama+Canon; tweet (14/N)].
A unified linear-attention testbed standardizes GLA → GLA5 (GLA + low-rank gating + alignment), GDN → GDN2 (GDN + low-rank gating), and Mamba2 at matched depth, hidden size, recurrent memory, and data pipeline — eliminating the hidden-parameter-count and hidden-kv-design advantages that plagued prior linear-vs-Transformer comparisons [Part 4.2 §Linear Models Testbed; tweet (15/N)].
Canon boosts GLA5 by ~20% token savings on MMLU; GLA5+Canon ≈ GDN2+Canon, and both outperform Mamba2+Canon — so the inter-linear-model differences collapse with Canon, exactly as the synthetic playground predicted [Part 4.2 §GLA5 vs GDN2; tweet (16/N)].
The gains from Canon are largest for the architectures with the weakest horizontal mixing: ranking by Canon-benefit gives Llama ≈ GLA > GDN > Mamba2, mirroring the Part 4.1 synthetic-task ranking and validating “horizontal mixing is the load-bearing primitive” at real-pretraining scale [Part 4.2 §Canon Benefits; tweet (17/N)].
Linear models look competitive on lm_eval-style aggregate benchmarks but collapse on retrieval: with 5 facts in context, linear ≪ Transformer at ~100 tokens; at 4k tokens linear models drop to random-guess accuracy [Part 4.2 §Retrieval; tweet (18/N)]. This isolates retrieval as the cleanest signal where the architectural gap survives.
The retrieval failure of linear models is not a memory-capacity problem (each layer’s recurrent state is over-provisioned) but a compression / retrieval-dynamics problem — so until that is solved, hybrid attention + linear-attention models remain the only viable scaling path [Part 4.2 §Retrieval Bottleneck; tweet (18/N)].
2-hop reasoning never emerges even at 8B / 1T tokens (L=0 accuracy ≈ random) — implying current pretraining pipelines add multi-hop supervision too late, and the architecture community has been chasing differences that are masked by this floor [Part 4.2 §Limits; tweet (19/N)].
Code, 16 Llama / LlamaCanon checkpoints, and 48 linear-attention checkpoints are released under Apache 2.0 (with the Lingua-fork modifications under BSD-3) at the PhysicsLM4 GitHub and HuggingFace, making the full architectural ablation reproducible end-to-end [project page §Release].

Method

Part 4.2 is structured as a controlled grid, not a single comparison. The Llama track holds the trainer (a fork of Meta’s Lingua with Canon-A/B/C/D layers, QK-Norm, z-loss, and partial RoPE added) constant and varies only the architecture (Llama vs LlamaCanon), model size (1.3B / 3B / 8B), and peak LR (0.002 / 0.003 / 0.005) — 16 checkpoints in total. The linear track adds a parallel apps.gla.train entry point and runs three linear-attention bases (GLA5, GDN2, Mamba2) each with and without Canon, against 18 Llama controls — 48 + 18 = 66 checkpoints. Every checkpoint pretrains on the same 1–2T token sequence of the Nemo corpus with the same batch size, schedule, and random seed up to the architecture-specific RNG draws.

Evaluation is deliberately wider than the in-house synthetic playground: MMLU, lm_eval suites, multi-hop birth-year retrieval, a modified Babilong long-context protocol, and length-generalization sweeps. Training-time MMLU curves are checked into the repository as interactive plots so Canon-vs-baseline comparisons can be replayed at any training step, not just at the endpoint. The GLA5 / GDN2 “alignment” is a non-trivial engineering step: it re-derives both designs in a unified kv format so that recurrent-memory budget can be matched exactly across them and against Mamba2, eliminating the “secret extra parameters” confound common in prior linear-model comparisons.

The Canon insertion is identical to Part 4.1 — 4-tap 1-D conv with residual wrapping — but applied at 1B–8B scale; the paper reports that Canon-A/B/C/D contribute cumulatively at this scale just as they did in the synthetic playground, and that the residual form is critical to stability when pretraining duration grows to 1T+ tokens.

Results

Llama vs LlamaCanon (1.3B–8B, 1–2T tokens): LlamaCanon strictly Pareto-improves Llama on MMLU across the LR sweep; the gap is largest at 1.3B (where reasoning ceilings bind) and narrows but persists at 8B [Part 4.2 §Llama+Canon results, training-time charts in the repo].
GLA5+Canon ≈ GDN2+Canon > Mamba2+Canon: on aggregate benchmarks, training curves of GLA5+Canon and GDN2+Canon become nearly identical once Canon is added, while Mamba2+Canon lags despite its hidden conv1d [Part 4.2 §Linear track; tweet (16/N–17/N)].
Token efficiency on MMLU: GLA5 → GLA5+Canon saves ~20% tokens to reach matched MMLU [Part 4.2 §GLA5; tweet (16/N)].
Retrieval scaling: with 5 facts injected at varying context lengths, Transformers stay near ceiling while linear models drop sharply with context length, hitting random-guess at 4k [Part 4.2 §Retrieval; tweet (18/N)].
Long-context generalization: NoPE and reduced-RoPE configurations outperform full RoPE when paired with Canon at long-context evaluation, replicating the Part 4.1 synthetic-task finding at the 8B / 1T scale [Part 4.2 §Long context; tweet (11/N)].
2-hop reasoning: zero-emergence at 8B / 1T tokens — every architecture tested sits at random-guess accuracy on the simplest 2-hop birth-year inference, even at 100-token context [Part 4.2 §Limits; tweet (19/N)].

Why it’s interesting

Part 4.2 is the empirical bridge that closes the synthetic-vs-real gap raised by Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers: every architectural ranking the synthetic playground predicted survives at 1–8B / 1–2T-token controlled pretraining, including the most surprising one — that Canon collapses the inter-linear-model differences (GLA5 ≈ GDN2 > Mamba2 with Canon) that the field has been spending years optimizing. The 2-hop-never-emerges result is the kind of clean negative finding that complicates Souper-Model: How Simple Arithmetic Unlocks State-of-the-Art LLM Performance and Incorrect Baseline Evaluations Call Into Question Recent LLM-RL Claims‘s skepticism about benchmark-driven architecture research with concrete evidence: it isn’t just that the benchmarks are noisy, it is that the capability they purport to measure doesn’t exist in any architecture at academic-and-bigger scale. The retrieval failure of linear models being a dynamics (not memory-size) bottleneck connects to the wiki’s existing MSA: Memory Sparse Attention for Efficient End-to-End Memory Model Scaling to 100M Tokens and Radial Attention: O(n log n) Sparse Attention with Energy Decay for Long Video Generation threads: solving long-context retrieval with linear models needs work on update / compression mechanics, not bigger recurrent states. And the 16+48 released checkpoints make this one of the few publicly reproducible architecture-ablation grids at the 1–8B / 1T-token scale — a counterweight to closed industrial reports.