Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers

TL;DR

Part 4.1 (v2.0) of the Physics of Language Models series argues that architecture comparisons at academic-scale pretraining (1.3B / 100B tokens) are dominated by noise and cannot reliably rank designs, so it replaces them with five controlled synthetic pretraining tasks — Depo, Brevo, Capo, Mano, Lano — that isolate reasoning depth, reasoning breadth, knowledge capacity, knowledge manipulation, and hierarchical-structure inference. In this playground the paper discovers Canon layers: a lightweight horizontal-residual block (a 4-tap 1-D convolution over neighboring tokens) that can be inserted at four positions (Canon-A/B/C/D) around any attention or MLP sublayer in any sequence model. Canon raises reasoning depth 2–4×, doubles linear-attention reasoning breadth, lifts NoPE to RoPE-level, makes GLA+Canon match Mamba2/GDN, and reveals that most of Mamba2’s advantage is its built-in conv1d (a partial Canon-B). At academic scale these synthetic predictions hold in noisy 1.3B/100B real pretraining; the companion Part 4.2 extends the same recipe to 1–8B / 1–2T-token runs.

Key claims

Academic-scale (1.3B / 100B tokens) pretraining is statistically unreliable for architecture comparisons because random initialization and data-shuffling alone cause 2–4% benchmark swings, and 2-hop reasoning fails to emerge at all — masking the architectural differences in deeper cognitive skills [§1, Fig. 1].
Five synthetic pretraining tasks — Depo (reasoning depth), Brevo (reasoning breadth), Capo (knowledge capacity in bits/parameter), Mano (knowledge manipulation), Lano (hierarchical-language CFG) — isolate atomic skills under controlled conditions, with vocabulary, context length (4096 tokens), and difficulty distributions explicitly engineered to avoid grokking instability [§2, §2.1].
An initial four-architecture comparison (Llama, GLA, Mamba2, GDN, all size-matched at 12L768D / 12L512D / 8L768D / 8L512D) shows linear-attention GLA is weakest overall, Mamba2 dominates knowledge but lags on reasoning, and Transformers + GDN dominate complex reasoning — even before any architectural intervention [§3.1, Fig. 4].
Canon layers are 4-tap trainable 1-D convolutions over neighboring tokens, inserted at four positions (Canon-A before attention, Canon-B inside attention, Canon-C before MLP, Canon-D inside MLP); the residual form h + Canon(h) is the load-bearing detail that keeps training stable without tuning [§4].
In Transformers, Canon improves reasoning depth by 200–400%, reasoning breadth by 30%, and knowledge-manipulation length by 30%, with minimal compute overhead; Canon-A/B/C/D contribute cumulatively and independently of whether the surrounding sublayer is attention, MLP, or MoE [§5].
Canon recovers in gated MLP and MoE designs the knowledge capacity lost relative to vanilla MLPs, indicating part of MoE’s underperformance on factual tasks is a training-efficiency problem that horizontal mixing fixes, not a capacity problem [§5].
Canon lifts weak architectures: NoPE+Canon matches (and sometimes exceeds) RoPE, and Canon lets the paper reduce the RoPE fraction while preserving long-context behavior — the first filed claim that horizontal mixing can substitute for positional encoding without giving up long-context generalization [§5, §8].
Across all linear models (GLA, Mamba2, GDN) Canon raises reasoning capability: GLA+Canon goes from 1-hop to 4-hop reasoning depth, doubles breadth and knowledge-manipulation length, and even surpasses Mamba2; most performance differences across linear architectures collapse once full Canon is added [§6].
Mamba2’s hidden conv1d is doing most of its work, not the SSM: removing the conv1d drops Mamba2 to GLA level, while replacing conv1d with full Canon surpasses original Mamba2 — i.e. “Mamba is mostly conv1d + linear attention,” and horizontal mixing is the load-bearing primitive [§6].
GDN benefits least from Canon because its gating and delta updates already capture some Canon-like behavior; this is presented as evidence that recent linear-model “refinements” may largely be re-deriving Canon under different names [§6].
Under a fully controlled apples-to-apples comparison (same depth, size, data, pipeline, with linear models given equal recurrent memory via the improved GLA→GLA5 and GDN→GDN2), linear models have ~40% higher knowledge capacity but Transformers retain 2–4× greater reasoning depth and stronger structural reasoning [§7].
The cause of linear models’ shallow reasoning is not insufficient memory (each layer’s recurrent state is over-provisioned) but cumulative compression / retrieval errors — pinpointing memory dynamics, not memory size, as the bottleneck [§7].
Real-life 1.3B / 100B-token pretraining results mirror the synthetic predictions where signal is detectable: Canon lifts NoPE to RoPE level and GLA to Mamba2/GDN level; removing Mamba2’s conv1d drops it to GLA level; linear models lag full Transformers on retrieval-heavy tasks; and all models fail 2-hop reasoning even at 100-token contexts [§8].

Method

The synthetic playground is the heart of Part 4.1. Each task is a sequence-prediction problem on a fully-controlled vocabulary with a closed-form ground truth, so accuracy can be evaluated without LM-as-judge or human annotation, and so the task can be sized to exactly the regime where the architecture’s bottleneck is exposed. Depo and Brevo evaluate in-context reasoning over directed permutations and DAGs (multi-hop traversal, topological-sort generation); Capo uses the bioS / bioR fake-biography generators from Part 3 to measure bits-per-parameter knowledge capacity in an undertrained regime (each biography exposed 100 times); Mano uses modular-arithmetic compositions to test mental knowledge manipulation; Lano uses context-free grammars with token-level ambiguity to require dynamic-programming-style structural reasoning.

Canon layers themselves are simple. Each Canon block is a 1-D convolution of kernel size 4 over token representations (trainable but tiny — ~ $d^2/4$ parameters per insertion point in the dense path), wrapped in a residual: h ← h + Canon(h). They are inserted at up to four positions per transformer block (Canon-A/B/C/D), and the same construction is dropped into GLA, Mamba2, and GDN with no changes to the underlying sub-layer. The paper repeatedly emphasizes that the residual form is what makes the addition stable across architectures — without it, Canon-only blocks degrade training.

The architectural ablations control for size by aligning total parameters across architectures (with the linear models getting both the standard MoE-free variants and the larger mlp variants), and aligning training pipeline by sharing batch size, schedule, LR, and random seed across all four architectures so that every model pretrains on identical token sequences. The Capo task widens this to a model-size sweep used to measure bit-per-parameter capacity across the (architecture, size) grid.

The companion code and 16 controlled Llama checkpoints (1.3B / 3B / 8B at three LRs on the Nemo corpus, 1–2T tokens) are released as PhysicsLM4 — Canon Layers code release (Physics of Language Models Part 4) and pair the synthetic-side findings here with the real-pretraining-side findings of Part 4.2.

Results

Reasoning depth (Depo): Transformers + Canon hit 2–4× deeper reasoning than Transformers alone at the same parameter count; GLA + Canon goes from 1-hop (untrained linear-attention) to 4-hop, matching Transformers without Canon [§5, §6, Fig. 5].
Reasoning breadth (Brevo): Canon adds ~30% breadth to Transformers and ~100% to GLA; the residual form is required for the gain to materialize without training instability [§5, §6].
Knowledge capacity (Capo): Linear models with Canon recover ~40% more bit-per-parameter than Transformers in the undertrained regime, but the gap closes under longer training — the persistent gap is in reasoning, not capacity [§7].
Mamba2 ablation: Replacing Mamba2’s built-in conv1d with full Canon (Canon-A/B/C/D) yields a model that surpasses original Mamba2; removing conv1d entirely drops it to GLA level — the conv1d is approximately Canon-B, and Canon-B alone explains most of Mamba2’s reasoning advantage over GLA [§6].
Positional encoding: NoPE + Canon matches RoPE on the synthetic playground and on Lano structural reasoning; partial RoPE + Canon is the best long-context configuration in the academic-scale real-life pretraining [§5, §8].
Real-life 1.3B / 100B: Canon-equipped GLA / NoPE close the gap to Mamba2/GDN and RoPE respectively on MMLU and retrieval; linear models still lag full Transformers on multi-fact retrieval (≪ Transformer at ~100 tokens, ≈ random at 4k tokens), even with Canon [§8].
2-hop reasoning floor: Even at 1.3B / 100B with the best architecture, no model exits random-guess on 2-hop reasoning over short (100-token) contexts — a finding the paper uses to argue 2-hop training supervision must begin earlier than current pipelines [§8].

Why it’s interesting

Part 4.1 reframes a long-running question on the wiki — how do architecture choices interact with training dynamics at scale? — by replacing the noisy academic-scale benchmarks with a controlled synthetic playground that exposes architectural strengths long before “emergence.” This sits directly alongside PhysicsLM4 — Canon Layers code release (Physics of Language Models Part 4) (the code+weights release the tweet announced as the companion to Part 4.2) and the broader Training stability at scale cluster — Canon is the same flavor of “tiny residual add that has outsized effect on signal flow” that QK-Clip (QK-Clip: Taking Muon Further on the Scaleup Journey), attention residuals (Attention Residuals (AttnRes / Block AttnRes) — Technical Report), and gated attention (A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training) all instantiate. The Mamba2 ablation is the sharpest claim: most of what distinguishes modern linear-model architectures from each other is recoverable by a 4-tap convolution, which both validates Mamba-3: Improved Sequence Modeling using State Space Principles‘s parallel ablation track and complicates Multi-head Temporal Latent Attention-style claims that fancier linear-attention variants intrinsically outperform vanilla GLA. Finally, the “2-hop reasoning never emerges at academic scale” finding is the kind of negative result that directly bears on Incorrect Baseline Evaluations Call Into Question Recent LLM-RL Claims and recent skepticism about benchmark-driven architecture research.