PhysicsLM4 — Canon Layers code release (Physics of Language Models Part 4)

TL;DR

Official code, training recipes, datasets, and 16 + 48 model weights for Physics of Language Models Part 4.1 (NeurIPS ‘25) and the in-progress Part 4.2, by Zeyuan Allen-Zhu (Meta). The release operationalizes Canon layers — a small architectural addition tested on synthetic pretraining playgrounds (Depo / Brevo / Capo / Mano / Lano) and then validated in a “real-life” pretraining setting at 1B / 3B / 8B Llama scale with up to 2T tokens. Companion fork of Meta’s Lingua trainer adds Canon-ABCD layers, QK-norm, z-loss, partial RoPE, and a parallel linear-attention track covering GLA5, GDN2, and Mamba2 with low-rank gating. Everything is Apache 2.0 (Lingua modifications stay BSD-3).

Key claims

The release reproduces results from Part 4.1 Architecture Design and the Magic of Canon Layers and supplies the apparatus for the forthcoming Part 4.2 on whether synthetic-pretraining gains transfer to real pretraining [README §Repository Contents].
16 Llama / LlamaCanon checkpoints span 1B / 3B / 8B, trained on the Nemo corpus for 1T or 2T tokens at peak LRs 0.002, 0.003, 0.005 — a controlled grid where the only architectural delta is Canon layers [README §canon_llama_recipes].
A linear-attention track ships 48 weights covering GLA5 (GLA + low-rank gating), GDN2 (GDN + low-rank gating), and Mamba2, each with and without Canon layers, against 18 Llama controls [README §canon_linear_recipes, §canon_linear_results].
Evaluation pairs synthetic playgrounds (Depo / Brevo / Capo / Mano / Lano) with multi-hop birth-year retrieval and a lightly-modified Babilong protocol — i.e. claims of transfer are tested on widely-used long-context benchmarks rather than only on the in-house tasks [README §Data Generators].
The Lingua fork exposes Canon-ABCD layers, QK-norm, z-loss, partial RoPE, and load_from_lingua_state so HF LlamaCanon checkpoints round-trip with the training codebase [README §lingua_modified].
Training-time MMLU curves for the released models are checked in as interactive charts so Canon vs. non-Canon comparisons can be replayed step-by-step rather than read off final numbers [README §canon_llama_results].

Method

Canon layers are described in Part 4.1 (SSRN 5240330) as small auxiliary blocks added alongside the standard transformer stack — the repo’s “Canon-ABCD” and “Canon-AbCD” naming suggests four insertion points (A/B/C/D) for the dense Llama path and three for the linear-attention path. The release does not re-derive the math; it ships them as configurable options in the Lingua fork. Reproduction is a one-liner: python -m lingua.stool script=apps.main.train nodes=1 config=apps/main/configs/canon_1B.yaml, with parallel apps.gla.train entry points for the GLA / GDN / Mamba2 variants.

The synthetic data side packages five playgrounds. Three (Depo, Brevo, Mano) are released here in full; Capo wraps the bioS and bioR generators from Part 3 (Knowledge Storage and Capacity Scaling Laws); Lano is the hierarchical-language-structure generator from Part 1. The multi-hop birth-year task and a Babilong few-shot variant supply the real-world evaluation harness.

Results

This is a code+weights release, not a paper, so headline numbers belong to Part 4.1 itself. The repository contributes the evidence trail: 16 Llama checkpoints in a controlled Canon-vs-baseline grid plus 48 linear-attention checkpoints, with training-time MMLU charts and a side-by-side comparison against open-source models trained in “realistic pretraining setups.” Part 4.2 will publish the synthesized story; the repo lets external groups verify or refute the Canon ablations directly.

Why it’s interesting

Allen-Zhu’s Physics of Language Models series is one of the few research programs that runs controlled, paired-comparison experiments at multi-billion-parameter pretraining scale, and Part 4 is the architecture-ablation chapter. The Canon layers result is the kind of small, additive modification that the wiki’s Training stability at scale cluster keeps catching — siblings to QK-Clip (QK-Clip: Taking Muon Further on the Scaleup Journey), attention residuals (Attention Residuals (AttnRes / Block AttnRes) — Technical Report), the spike/sink/sparse analyses (The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks, A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training), and the broader μP scaling line (Spectral Condition for μP under Width-Depth Scaling, Beyond MuP 3: Special Cases, Special Treatment (Embedding, LM Head, RMS Norm)). It also lands close to Generalization at the Edge of Stability and Test-Time Scaling Makes Overtraining Compute-Optimal in caring about what happens when you actually train long, and to Mamba-3: Improved Sequence Modeling using State Space Principles in carrying both an attention and a state-space track in the same study. The synthetic→real transfer question is the same one The Well: 15TB of Physics Simulations and Data Darwinism Part I: Unlocking the Value of Scientific Data for Pre-training raise on the data side; here it is being raised on the architecture side.