Skip to content
Bud's Wiki

Training stability at scale

TL;DR

Section titled “TL;DR”

A late-2025/early-2026 cluster of papers locates large-model training stability in architectural choices (normalization variants, residual-mixing constraints, gated rescaling) rather than in parametrization recipes like µP. The shared empirical claim is that modern normalization (QK-Norm, RMSNorm, GatedNorm) and constrained residual geometry (Birkhoff-polytope residual mixing) subsume what classical parametrization tricks were originally designed to fix — and in some cases the architectural fix is strictly better because it also removes pathologies (outlier activations, identity-mapping violations across depth) that the parametrization fix never addressed.

Key claims

Section titled “Key claims”
  • Under modern architectures, the attention-logit blow-up that µP was designed to suppress simply does not occur under standard parametrization; ablating QK-Norm restores the instability µP reports, so the stability is doing the work, not the parametrization (How to Set the Learning Rate for Large-Scale Pre-training? §6.3, Figs. 8–9).
    • Conflict: Controlled LLM Training on Spectral Sphere argues the opposite resolution — µP’s motivation still stands, but current optimizers (including Muon) are only “half-aligned” (they control update spectra but allow weight spectra to drift), and the right fix is to constrain both weights and updates to a spectral sphere rather than rely on QK-Norm to mask the symptom (Controlled LLM Training on Spectral Sphere README §1, Abstract). The two papers prescribe directly opposing actions: drop µP and lean on normalization (2601.05049) vs. enforce µP harder via the optimizer (2601.08393).
    • Finetuning-side replication of the parametrization position: Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning shows that for LoRA finetuning, µP-style asymptotic analysis (extended to the joint n,r→∞n, r \to \infty limit as µA) still yields tight, predictive scaling rules for the optimal LR — including a configuration (Init[B], α=1\alpha=1) where LoRA’s optimal LR transfers directly to FFT (Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning §4.2, Fig. 5). The µP framework is not dead in the finetuning regime even if its motivating instability has been suppressed in from-scratch pretraining.
    • Vision-side replication: ViT-5: Vision Transformers for The Mid-2020s reports that adding QK-Norm to ViTs gives a modest accuracy bump and a large reduction in sharp loss spikes during optimization — the same spike-suppression role QK-Norm plays in LLMs (ViT-5: Vision Transformers for The Mid-2020s §3.6, Fig. 5).
  • Module-specific learning rates (a µP prescription) yield essentially no measurable improvement over a single global LR at 4B-MoE scale under modern normalization, because per-module AdamW update magnitudes are already uniform across layers (≈0.2) consistent with rotational equilibrium (How to Set the Learning Rate for Large-Scale Pre-training? §6.1–6.2, Figs. 5–7).
  • Attention sinks and residual sinks both arise from the same functional pressure — outliers paired with softmax/RMSNorm act as implicit rescale factors for non-outlier components, which is load-bearing for training stability (removing normalization or clipping outliers both degrade training) (A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training §3, Abstract conclusions (1)–(2)).
  • The implicit rescaling can be replaced with explicit learned gating (Gated Attention, GatedNorm), giving ≈+2-point average pretraining gain and ≈1.2-point less W4A4 quantization degradation — i.e. architecture choices that maintain stability and produce low-precision-friendly activation distributions (A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training Abstract conclusion (3), Table 1).
  • Unconstrained Hyper-Connections break the identity-mapping property of residual stacks at scale, producing forward/backward signal gains up to ~3000× and visible 27B-scale loss spikes; constraining the residual mixing matrix to doubly stochastic matrices (Birkhoff polytope) via a differentiable Sinkhorn projection bounds composite gain to ~1.6× across arbitrary depth (mHC: Manifold-Constrained Hyper-Connections §3.1, §4.1, §5.4, Figs. 2/3/7).
  • Architectural stability fixes hold across compute and data scaling — the loss-improvement gap of mHC over baseline is maintained from 3B → 9B → 27B, and across a 1T-token run at 3B (mHC: Manifold-Constrained Hyper-Connections §5.3, Fig. 6).
  • The slope of LM pre-training scaling laws — not just the intercept — appears to be set by a structural primitive (softmax + cross-entropy on peaked target distributions), yielding a universal 1/3 time-scaling exponent that interventions like optimizer choice, data mixes, and most architecture tweaks do not change (Universal One-third Time Scaling in Learning Peaked Distributions Abstract, Figs. 1–3). This is the “tyranny of the slope” framing: stability/architecture interventions move the intercept; only loss/output-head changes can move the slope.
  • Constraining both weight matrices and updates to a per-module spectral sphere of radius R=Θ(dout/din)R = \Theta(\sqrt{d_\text{out}/d_\text{in}}) — the steepest-descent direction under the spectral norm, with hard or dynamic retraction — outperforms AdamW and Muon on Dense 1.7B, MoE 8B-A1B, and a 200-layer DeepNet while suppressing outliers, bounding activations, and improving MoE router load-balancing (Controlled LLM Training on Spectral Sphere README §1, §2, §4). This is the optimizer-side analogue of the architecture-side mHC residual-mixing constraint: same manifold-constraint recipe, applied to a different stability locus.
  • Architectural stability components do not transfer trivially across modalities: porting SwiGLU MLPs (a standard LLM choice) into ViTs alongside LayerScale produces over-gating — both act as channel-wise filters, the combination drives intermediate representations to excessive sparsity, and accuracy drops (84.16% → 83.70% at ViT-B, FID 14.57 → 16.22) (ViT-5: Vision Transformers for The Mid-2020s §3.3, Table 2). The QK-Norm, RMSNorm, LayerScale, and bias-free-QKV pieces do transfer; SwiGLU does not.
  • MoE router stability has a parallel “remove the tuning axis” story: Quantile Balancing (QB) reformulates per-step expert assignment as the linear program “find the bias vector that would have produced balanced load last step” and solves it via alternating quantile iteration — eliminating the aux-loss coefficient, the fixed-bias-update magnitude, and the momentum coefficient that prior recipes (GShard, DeepSeek loss-free, Arcee SMEBU) all required. Validated at 32B-A5B / 1e22 FLOPs / 326B tokens on Marin with zero loss spikes and no leading dense layers (Quantile Balancing: A Hyperparameter-Free MoE Load Balancing Method).

Recent contributions

Section titled “Recent contributions”
  • [2026-08-31] Sliding-window beats linear attention: Practical corollary of the “sinks are load-bearing” thesis (A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training, StreamingLLM and the Discovery of Attention Sinks): permanently-retained attention sinks under a sliding-window mask are enough to preserve long-context behavior of a pretrained LLM at inference without any post-training, and beat linear-attention retrofits by 2–10× on NIAH/BABILong. Sink-preservation as an inference-time knob, not just a training-time architectural choice.
  • [2026-08-28] OpenDM — DM0.5: An Open-World Foundation Model for General-Purpose Embodied Intelligence: DM0.5 (Dexmal) uses separate learning-rate groups for the VLM backbone and the Action Expert — VLM backbone trained at a smaller LR to reduce catastrophic forgetting and preserve general vision-language capabilities during action co-training. Simpler-than-Anchor-Align answer (a knob rather than a distillation loss) to the same “grounding wall” failure mode Pretrained to Imagine, Fine-Tuned to Act: The Rise of World-Action Models VLM2VLA and Anchor-Align: Generalizable VLA Finetuning via Representation Anchoring and Language-Action Alignment Anchor-Align target. Combined with an internal automatic VL data generation pipeline (image + video + visual instruction) so the VLM branch is both slowed and fed during co-training rather than only slowed.
  • [2026-08-26] Qwen3.8-Flash-Next — Hybrid Attention with QSA, Gated Residual, N-gram Embedding: Qwen3.8-Flash-Next introduces Gated Residual — 4-branch widened residual streams with rank-320 bottlenecks, modulated by an element-wise data-dependent read gate and per-branch scalar write gate. Positioned as a residual-stream surgery in the AttnRes / mHC / QK-Norm family: preserves training stability while adding finer-grained cross-layer expressiveness, with inference overhead kept low.
  • [2026-08-25] Mitigate Silent Expert Death In Ultra-Sparse MoE: Adds an auxiliary-loss / critical-window locus of stability distinct from architecture, optimizer, and parametrization. In ultra-sparse MoE (top-8-of-768 at 60B/180B), lower-layer routed experts silently die: weight norms fall to ~1/10 of init while loss and load-balance dashboards look normal, and masking them barely changes quality. Diagnoses the mechanism as shallow-expert AdamW second-moment falling ~3 orders of magnitude below deeper layers — well below the conventional ϵ=10−8\epsilon=10^{-8}, which then caps the adaptive update while decoupled WD keeps shrinking parameters. Shows Muon’s ϵ=10−7\epsilon=10^{-7} (Frobenius-norm scale) is comparable to AdamW ϵ ⁣≈ ⁣10−11\epsilon\!\approx\!10^{-11} (per-coordinate), a mechanistic complement to Controlled LLM Training on Spectral Sphere‘s “Muon is half-aligned with µP” scale argument. Numerical fixes rescue norms but not downstream tasks — motivating LLAL (auxiliary LM-head loss on an early MoE layer during the first ~3% of training), which prevents collapse and persists after removal, yielding +2.8 MMLU / +4.8 MMLU-Pro at 180B.
  • [2026-08-20] Marin scaling laws simulate the entire training trajectory of a 67B MoE run within 1%: Marin’s 67B MoE run hits target loss within 1% and matches predicted per-step loss within 1% — implicit evidence that the Delphi + Quantile Balancing + AdamH stack suppresses the transient instabilities that typically break industrial MoE runs; the same ladder is now forecasting a 535B MoE trajectory in flight.
  • [2026-08-20] GEN-0: Embodied Foundation Models That Scale with Physical Interaction: GEN-0 (Generalist AI, Nov 2025) reports LLM-style ossification in robotic policy pretraining at ~O(1B) parameters — smaller models lose the ability to absorb new interaction data over the course of training, while 7B+ models continue to improve. First filed observation of the phenomenon in the embodied-model regime and at three orders of magnitude larger scale than prior LLM literature (O(10M)), positioned as evidence that stability thresholds for physical commonsense sit at a higher compute activation threshold than for abstract reasoning.
  • [2026-08-17] Training Dyna-2 at million-hour scale, repeatably: Dyna Robotics’s infrastructure companion to the 1M-hour Dyna-2 run supplies a concrete production distributed-Muon recipe for the page’s “framework-default Muon lacks FSDP/HSDP support” open question: topology-aware sharding that picks per-node NVLink hybrid sharding (with node-level duplication) above a node-count threshold and world-sharding below it, reporting ~3× speedup at scale over world-sharding. The mean-vs-median measurement (fully sharded looks competitive at median but is 2.6× worse on the mean step time due to 7.6× broadcast traffic on InfiniBand vs NVLink, and synchronous training pays wall-clock as the sum of all steps) is the specific quantification the distributed-Muon thread has been circling.
  • [2026-08-16] Cracks in the Foundation: Seemingly Minor Architectural Choices Impact Long Context Extension: Bertsch et al. (CMU / Ai2) add a long-context-extensibility locus to the architecture-side stability framing: four “minor” dense-Transformer choices — normalization variant, GQA, pretraining context length, sliding-window attention — each drawn from an Olmo/Llama/Qwen production decision, individually barely move short-context loss but combine compoundingly negatively, dropping long-context downstream performance up to 47% when 3+ combine. Short-context loss and validation are blind to these differences. The OlmPool release (26 comparable 7B models, 170k+ GPU-hours, pre/post-extension checkpoints) contains configurations that beat Llama 3 on long-context extensibility, and attention-sink and attention-distribution analyses attribute specific long-context degradations to specific choices — a mechanistic follow-on to A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training‘s “sinks are load-bearing rescaling” claim. Practical: apply context extension early in pretraining as a cheap long-context-extensibility diagnostic.
  • [2026-08-14] Breaking the Softmax Bottleneck: A High-Rank RNN Language Model: Backfill of the seminal “Softmax bottleneck” paper (Yang, Dai, Salakhutdinov & Cohen, 2017) that both Lost in Backpropagation: The LM Head is a Gradient Bottleneck and Universal One-third Time Scaling in Learning Peaked Distributions build on. Formalizes LM as matrix factorization: log-probability matrix has rank ≤ embedding dim dd, so if the true matrix is high-rank (as natural language is hypothesized to be), Softmax with d≪∣V∣d \ll |V| cannot represent it. Mixture of Softmaxes fixes the expressivity side; the 2026 papers show the same dd-vs-VV mismatch also produces optimization (rank-dd Jacobian) and scaling-law-slope (1/3 exponent) pathologies at the same layer. Historical anchor for the “LM head as structural primitive” framing on this page.
  • [2026-08-13] Dion3: Full-Stack Orthogonal Updates: Dion3 (Amsel, Zhang, Ahn, Naeimi, Feng, Chen, Dao, Langford) consolidates three separately-filed lines of Muon-overhead work into one drop-in optimizer: Gram Newton-Schulz for algorithmic FLOP reduction, CuteDSL symmetric-GEMM kernels, megabatching for communication overhead, and a new row-selection update rule that strictly improves on Dion/Dion2’s compression schemes. Claims up to 6× optimizer step-time reduction over Muon while matching or improving loss — the strongest end-to-end “distributed Muon actually affordable” datapoint on this page yet, and the convergence point of the Gram Newton-Schulz: A Fast, Hardware-Aware Newton-Schulz Algorithm for Muon and microsoft/dion — distributed Muon, Dion2, Dion, and NorMuon orthonormal optimizers threads.
  • [2026-08-12] MAGI-2 Preview: Scaling Video Generation Models Efficiently: Sand.ai MagiMuon extends the Muon-family optimizer to head × expert matrix batches from Ultra-Fine-Grained MoE: router gates batched by head, expert up/gate/down projections flatten head-and-expert into the batch dimension, Muon orthogonalization runs independently per matrix with the matrix batch distributed across ranks. Adam is retained for adapters, mHC, attention sinks, and gating parameters. Concurrent with Kimi K3’s Per-Head Muon (attention Q/K/V head split); different structures, same principle — explicit head structure in the model should be reflected in the optimizer.
  • [2026-08-11] Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization: Formal Hyperball paper (Wen, Dang, Lyu, Ma, Liang) — the arxiv writeup of what MuonH (hyperball) sets new modded-NanoGPT speedrun record at 3325 steps — Keller Jordan announcing @wen_kaiyue's result announced and Delphi: Scaling Laws That Extrapolate 300× Past the Fit deployed. Wraps any base optimizer (Adam, Muon, NorMuon, KL-SOAP) with Frobenius-sphere projection on weights + unit-Frobenius updates; on 1B Qwen3 pretraining, MuonH gives 1.4–1.6× token-equivalent speedup over MuonW with the gap growing over training. On modded-nanogpt Track 3, AdamH/MuonH/NorMuonH/KL-SOAP-H all beat their WD counterparts, with KL-SOAP-H setting the fastest step-count on the leaderboard — the strongest optimizer-agnosticism claim yet. Theory (§4) derives the mechanism from rotational equilibrium: WD’s two coupled jobs (setting equilibrium radius ρ* and setting angular step η/ρ*) get severed by pinning ρ directly, so η becomes an explicit angular learning rate. Companion Slack observation from @wen_kaiyue: Muse Glimmer’s per-matrix weight RMS is pinned at ~6e-3 = 0.5/√6656 — suggests a frontier open release (Introducing Muse Glimmer: An Open Agentic Model That Runs on Your Device) may already be running Hyperball-shaped optimization in production.
  • [2026-08-06] Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks: Kosson, Messmer & Jaggi (2023) extend SMD (Spherical Motion Dynamics: Learning Dynamics of Neural Network with Normalization, Weight Decay, and SGD) from SGDM to AdamW / Adam+L2 / Lion and give the prescriptive half of the story: build “Rotational Variants” that project to the equilibrium ‖w‖ and enforce the equilibrium Δθ explicitly, dispensing with weight decay tuning. Central mechanistic claim: AdamW beats Adam+L2 (and Weight Standardization beats vanilla LayerNorm) because AdamW’s equilibrium rotation depends only on hyperparameters while Adam+L2’s depends on gradient magnitude — so identical HPs produce ~30× per-layer rotation variation (≈1000× effective-LR variation) in Adam+L2 but a tight band in AdamW. Direct theoretical ancestor of Modular Manifolds, Controlled LLM Training on Spectral Sphere, Delphi: Scaling Laws That Extrapolate 300× Past the Fit‘s AdamH, and Introducing Inkling — Thinking Machines' open-weights 975B/41B MoE multimodal reasoning model‘s WD∝LR² coupling. Caveat matching the filing pointer: RV matches baseline AdamW zero-shot on 5/6 configs — impressive as theory validation, not a benchmark win.
  • [2026-08-06] Spherical Motion Dynamics: Learning Dynamics of Neural Network with Normalization, Weight Decay, and SGD: Backfill of the SMD foundation the rest of the rotational-equilibrium / weight-geometry cluster on this page builds on. Wan, Zhu, Zhang & Sun (2020) prove that for scale-invariant weights under SGDM+WD the weight norm converges to an equilibrium at linear rate long before the loss converges, and derive the closed-form angular update Δθ ≈ √(2ηλ/(1-β)) that depends only on hyperparameters. This is the mechanistic root of How to Set the Learning Rate for Large-Scale Pre-training?‘s “per-module AdamW updates ≈0.2 across all layers” observation and of the whole SSO / AdamH / modular-manifolds / Inkling-WD∝LR² line — but Kyle’s team-side flag (per the filing pointer) is worth carrying forward: the framework studies angular updates in isolation and its practical gains in the original paper are equilibrium-recovery hacks (rescale ‖w‖ ← ‖w‖·√α at LR drops) rather than headline accuracy improvements.
  • [2026-08-05] CMuon: Accelerating and Stabilizing Diffusion Transformer Training via Chunked Momentum Orthogonalization: CMuon (Chen, Sun, Yuan) locates a DiT-specific Muon pathology that the LLM-side literature does not surface: standard DiTs fuse AdaLN (scale/shift/gate) and QKV projections into single tensors for GEMM efficiency, and Muon’s per-tensor Newton–Schulz orthogonalization treats the fused tensor as one operator — inducing implicit subspace coupling that distorts updates and produces vanilla Muon’s late-stage FID plateau on DiTs. Fix is a one-line decoupling: chunk the fused matrices along their functional boundaries before msign, orthogonalize each chunk independently, re-concatenate. Reports 675M DiT / ImageNet 256² → FID 1.18 in 200 epochs, >2× over AdamW, past vanilla Muon’s plateau. Third distinct Muon-pathology paper on the wiki alongside SSO (weight-spectrum half-alignment) and Aurora (row-leverage on rectangular matrices), and the first one whose failure mode is architecture-fusion-specific rather than optimizer-internal — reads as the principled version of Krea 2’s ad-hoc “exclude first/last linears + Nesterov” Muon workaround on MMDiT.
  • [2026-08-01] Multi-Head Attention Residuals: Multi-Head Attention Residuals (MHAR) is the multi-head lift of Moonshot’s Attention Residuals: reshaping the single per-layer routing query into H per-subspace heads so different feature subspaces can softmax-attend to different prior layers, with a U-shaped optimum at H=4 or H=8 across 100M–1B scales, zero extra parameters, and fused Triton kernels bringing training throughput from 0.2–0.5× to 0.55–0.88× of a plain-Transformer baseline. Delta-attention-residual mid-training at 8B is identity-preserving and yields +3.2 GSM8K / +3.1 GPQA — the AttnRes lineage now has a low-friction mid-training upgrade path.
  • [2026-07-30] A Defense of the Quadratic Model: Meterez, Nair, Morwani, Pehlevan, Kakade, Damian defend the local-quadratic model of LLM optimization: at 150M params / 3B tokens, Taylor-expanding the loss (quadratic) and model (linearized) at intermediate checkpoints reproduces the real training trajectory over windows lasting up to ~10% of training, with better agreement late in training and under cosine than under constant-LR. Deep Lanczos quadrature (m=1200) resolves the Hessian spectrum across six orders of magnitude, revealing a vocab-size head of eigenvalues (neural collapse) + a universal power-law tail that is invariant to batch size and to Adam preconditioning. Local linear-stability probes show LLM training operates at a stochastic EoS at small batch and a deterministic EoS at large batch, matching linear-stability theory — giving a data-side complement to the architecture/optimizer/parametrization loci already tracked here: whatever knobs you turn, the underlying loss is well-approximated by a structured quadratic and training sits on its stability boundary.
  • [2026-07-30] The Obsessed Encoder — Latent-Space Misallocation in JEPA-Style Self-Supervised Models: Enigma reports that the load-bearing anti-collapse regularizers of modern JEPA-style SSL (SIGReg’s Cramér-Wold + Epps-Pulley Gaussianity test in LeJEPA / LeWorldModel; DINOv3’s Sinkhorn-Knopp centering + KoLeo nearest-neighbor entropy) are silently defeated by low-entropy predictable features — an expressive encoder can hash a 3-DoF manifold into 192-D so every random 1-D projection matches N(0,1)\mathcal{N}(0,1), and KoLeo continues to spread neighbors apart in the hashed space. A stability-locus datapoint distinct from the architecture / optimizer / parametrization framings tracked on this page: Gaussianity-of-projections and nearest-neighbor entropy are not load-bearing collapse detectors when the encoder is expressive enough to hash a low-D signal, which suggests SSL anti-collapse mechanisms may need to bound intrinsic dimensionality rather than marginal shape.
  • [2026-07-28] Online KL Shampoo (OKLS) — Tilde Research's KL-optimal full-matrix AdaGrad approximation optimizer with 1.45× parameter efficiency over Muon: Tilde Research’s third 2026 optimizer, Online KL Shampoo (OKLS), attacks a different pathology than Aurora (row leverage) and Compositional Muon (QK/OV compositions): the history of gradient correlations that Muon deletes each step by re-orthogonalizing. Approximates full-matrix AdaGrad via KL-optimal Kronecker factors (novel vs Shampoo/Dion’s Frobenius-optimal factors) with zero-staleness inverse-square-root preconditioning; claims 1.45× parameter efficiency over Muon at 98% throughput across 200M–1B parameters. The three-way Tilde cluster (Aurora / CM / OKLS) now asks whether disjoint-pathology fixes compose or saturate the same optimizer headroom.
  • [2026-07-27] Delphi: Scaling Laws That Extrapolate 300× Past the Fit: Delphi’s AdamH optimizer intervention (Kaiyue Wen, Marin) is a stability lever framed as removing weight decay from the hyperparameter search: every Adam step is followed by a Frobenius-norm-sphere retraction that first takes a step of Frobenius norm matching η∥W∥F\eta \|W\|_F regardless of the raw update magnitude, then projects back to the sphere of radius ∥W∥F\|W\|_F so the weight norm is preserved exactly. There is nothing left for weight decay to regularize away, and in the Delphi width-scaling replication the best learning rate stays more bounded across widths under AdamH than under Adam. Sits alongside SSO (Controlled LLM Training on Spectral Sphere, per-module spectral sphere) and the Inkling weight-decay-∝-LR² coupling rule (Introducing Inkling — Thinking Machines' open-weights 975B/41B MoE multimodal reasoning model) as three flavors of the same idea: constrain the geometry of weights (Frobenius sphere / spectral sphere / LR-coupled decay) so the hyperparameter search is smaller and stability is enforced by construction rather than tuning.
  • [2026-07-24] SANA-Video 2.0: Hybrid Linear Attention with Attention Residuals for Efficient Video Generation: First external adoption of Moonshot’s Block Attention Residuals (Attention Residuals (AttnRes / Block AttnRes) — Technical Report) outside Kimi Linear — SANA-Video 2.0 uses AttnRes to route completed block summaries into later linear layers of a 5B/14B video DiT and reports the same PreNorm-dilution signature Moonshot documented on LLMs: ~12% lift in deep-layer effective rank. Confirms AttnRes generalizes from LLM stacks to diffusion transformers.
  • [2026-07-16] Kimi K3 — Open Frontier Intelligence (2.8T MoE with KDA + AttnRes): Kimi K3 pairs Per-Head Muon (Muon with per-attention-head decoupling) with QAT-from-SFT in MXFP4 weights + MXFP8 activations and reports stable training at 2.8T parameters with 16/896 expert sparsity.
  • [2026-07-15] Introducing Inkling — Thinking Machines' open-weights 975B/41B MoE multimodal reasoning model: Inkling’s public training recipe: hybrid Muon (matrix weights) + Adam (everything else) with weight decay coupled to LR² — the LR²-coupling is claimed to keep weight-matrix norms stable across training horizons, letting the same recipe transfer across LR schedules. First frontier-scale from-scratch pretraining run to name modular-manifolds-style spectral control + LR²-coupled decay as its stability primitive; 45T-token, 975B/41B MoE run on GB300 NVL72.
  • [2026-07-13] Modular Duality in Deep Learning: Bernstein & Newhouse’s foundational duality-map paper — the theoretical origin of the rectangular Newton-Schulz iteration that Muon uses to orthogonalize its momentum. Derives per-layer duality maps from operator norms (RMS→RMS for Linear, ∞→RMS for Embed, max-over-kernel for Conv2D) and composes them via the modular norm’s max-weighted composition, giving straightforward all-layer updates. Shows µP and Shampoo both fall out as partial approximations to the single √(d_out/d_in)·UVᵀ duality map for Linear — reframing “should we tune per-module LRs?” as “which pieces of this duality map does the optimizer actually implement?” Grounds the SSO claim that Muon is “half-aligned with µP” in a precise object.
  • [2026-07-10] The 4-bitter Lesson: Balancing Stability and Performance in NVFP4 RL: Dequantized backward as a new stability-under-quantization lever: replacing y = x·w_bf16 in the BF16 backward with y = x·DQ(Q(w_bf16)) matches the chain rule to the actual quantized forward, eliminating the grad-norm spikes that appear under the “quantized forward, higher-precision backward” baseline. Under Adam’s momentum + adaptive second moment the extra NVFP4 gradient variance is absorbed, yielding far fewer large spikes than BF16 backward. Combined with 4/6 and selective precision, NVFP4 RL matches BF16 grad-norm across five runs where any two-out-of-three combination still spikes.
  • [2026-07-09] Scaling with Collapse: Efficient and Predictable Training of LLM Families: Bergsma et al. deploy scaling-collapse residuals as a production stability signal on a Cerebras 300M–3.9B (Celerity) family: at 1.8B, collapse residuals surfaced a numerical instability at ~60% of training, whereas the raw loss trend only revealed it at ~90%. Ablations against the collapse reference isolated the bug to a microbatch-size-triggered loss-kernel numerical issue and enabled a targeted restart. Concrete demonstration that the Qiu et al. supercollapse diagnostic operationalizes at LLM scale as an earlier-and-sharper warning than trend-watching. Also documents CompleteP outperforming µP for joint width–depth scaling on this family, adding a data point to the µP-vs-CompleteP debate elsewhere on the page.
  • [2026-07-09] Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks: Qiu et al. establish scaling collapse and supercollapse as a live diagnostic: mis-parameterization (dropping µP for a constant LR across widths) breaks collapse even at small scales where final losses are indistinguishable. Adds a new class of stability signal to the page — collapse residuals give an earlier, more sensitive warning of scaling-parametrization pathologies than final-loss comparison or trend-watching alone. Complements Training stability at scale‘s architecture-vs-optimizer-vs-parametrization framing with a fourth “does dynamics stay consistent across scales?” axis.
  • [2026-07-08] LingBot-Video: Scaling Mixture-of-Experts Video Pretraining for Embodied Intelligence: LingBot-Video’s stability recipe for scaling a MoE video DiT to 120B/11B-active: QK-Norm (per-head RMSNorm on Q/K before attention), adaLN-single modulation (one shared timestep MLP + per-layer trainable modulation tables, zero-init on shared projection), auxiliary-loss-free bias for MoE load balancing plus sequence-wise Lseq to prevent per-video routing collapse, and √(d/d_a)-style variance-preserving expert init.
  • [2026-07-01] To Use or not to Use Muon: How Simplicity Bias in Optimizers Matters: Dragutinović et al. attack Muon from a new angle — not stability, not µP-alignment, but inductive bias. Under an exact-SVD, no-momentum abstraction of Muon (Spectral GD) on 2-layer deep linear networks, they prove GD’s classical saddle-to-saddle dynamics sequentially increase solution rank (largest singular value first) while Spectral GD learns every singular component in parallel at equal rate — losing GD’s implicit rank-based regularization. Two constructed head-to-heads have SGD beat Muon: (a) a multi-source “routing” task where SGD discovers a shared 4-D representation and generalizes to unseen source pairs while stochastic Spectral GD memorizes each pair despite matching train loss, and (b) MNIST + per-class spurious pixel where SGD’s dominant-first learning gives higher clean-digit accuracy up to spurious intensity ≈2. Also observes that after the theoretical low-rank modes are learned, Spectral GD’s discretization noise makes the update full-rank and it starts fitting that noise in a highly oscillatory trajectory that momentum does not damp — a mechanism-level hint that Muon may be less robust at high learning rates in richer settings than the current stability literature captures.
  • [2026-06-30] What do near-optimal learning rate schedules look like?: Naganuma et al. find that on the small WikiText-103 Transformer workload, the most aggressive near-optimal LR shapes produce some training-instability blowups (large-perplexity runs) that inflate confidence intervals, echoing the broader story that the optimal shape sits near the stability boundary — and on linear regression the analytical optimum is exactly at the edge-of-stability. AdamW weight decay (a known stability knob) is the strongest covariate of optimal shape, supporting the architecture/optimizer-side stability framing on this page.
  • [2026-06-24] Krea 2 Technical Report: Krea 2’s architecture-stability moves on a production T2I MMDiT: GQA + sigmoid-gated attention (small wall-clock cost, no quality gain but visibly smoother loss and gradient-norm curves), zero-centered RMSNorm with weight decay on its learnable params, QKNorm, per-block tunable bias replacing AdaLN MLPs. Reports Muon (Dion implementation + Moonlight RMS-matched HP transfer) is stable enough to beat AdamW at low and high resolution only after excluding the first and last linear layers and adding Nesterov momentum — otherwise it triggers loss/gradient-norm spikes.
  • [2026-05-26] Laguna M.1/XS.2 Technical Report: Laguna documents the M.1→XS.2 stability redo: keep the first Transformer layer dense, FP32 RMSNorm/RoPE, configurable gradient-reduction dtypes per collective (FP32 default for sensitive paths including the LM-head input-gradient all-reduce), and a WSD schedule with the peak chosen from an N,D scaling law rather than a one-shot cosine tune.
  • [2026-05-24] Addition of Muon optimizer to torch.optim — PyTorch team welcomes PR (Issue #148819): PyTorch maintainers’ July-2025 comment welcoming a PR to add Muon to torch.optim, explicitly citing Moonshot’s MuonClip / Kimi K2 deployment as the social proof. Marks the point at which Muon transitioned from “research-lab optimizer” to a framework default. The infrastructure caveat — distributed Muon (FSDP/HSDP) is still not supported in core, and Kimi K2 disables FP8 GEMM for the Newton–Schulz step — is the open piece the SSO line (Controlled LLM Training on Spectral Sphere) and Aurora (Aurora: A Leverage-Aware Optimizer for Rectangular Matrices) are addressing.
  • [2026-05-23] Quantile Balancing: A Hyperparameter-Free MoE Load Balancing Method: hyperparameter-free MoE load balancer (QB) from Jianlin Su, formulated as a per-step LP over bias offsets and solved via alternating quantile iteration. Sits in the same lineage as MuonClip / qk-clip — “eliminate the tuning knob entirely” — but applied to the router rather than to attention logits. Validated by the Marin team at 32B-A5B / 1e22 FLOPs (326B tokens) with zero loss spikes; 1e23-FLOP run underway. Suggests MoE routing is another locus where the “constrained optimization replaces hand-tuned heuristic” pattern works.
  • [2026-05-23] Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning: introduces µA (Maximal-Update Adaptation), the LoRA analogue of µP, characterizing optimal-LR scaling in the joint n,r→∞n, r \to \infty limit. Identifies Init[B] + α=1\alpha=1 as the unique LoRA configuration whose optimal LR coincides with full-finetuning’s, enabling direct LR transfer. Provides a finetuning-side counterpoint to 2601.05049: µP-style asymptotic analysis is still load-bearing here, even though from-scratch LLM pretraining no longer needs it under QK-Norm. Verified across SFT (5 models), RLVR, and Stable Diffusion.
  • [2026-05-23] ViT-5: Vision Transformers for The Mid-2020s: vision-side replication of the architectural-stability story — QK-Norm in ViTs suppresses sharp loss spikes during optimization, mirroring the LLM-side finding. Adds a cross-modality negative result: SwiGLU + LayerScale together over-gate ViTs (both are channel-wise filters; combined sparsity hurts), so the LLM→ViT port has to drop SwiGLU even though every other component transfers. Hits ImageNet-L/384² SoTA (86.0%) and SiT-XL FID 1.84.
  • [2026-05-23] Controlled LLM Training on Spectral Sphere: introduces SSO, an optimizer that constrains both weights and updates to a per-module spectral sphere via steepest descent under the spectral norm + Newton–Schulz msign + retraction. Argues Muon is “half-aligned” with µP (controls updates, not weights) and that closing this loop gives strict Θ(1)\Theta(1) activation control end-to-end. Outperforms AdamW and Muon on Dense 1.7B, MoE 8B-A1B, and 200-layer DeepNet; ships as a Megatron-LM fork with Atomic Module Sharding and cached singular vectors. Directly conflicts with 2601.05049 on whether µP’s motivation still stands.
  • [2026-05-23] Universal One-third Time Scaling in Learning Peaked Distributions: derives a universal 1/3 time-scaling exponent for cross-entropy loss when the target distribution is peaked, attributing it to softmax saturation and the resulting power-law-vanishing gradient. Reframes most pretraining interventions (Muon, data mixes, normalization) as intercept-only shifts of the scaling law — its slope is structural.
  • [2026-05-23] A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training: unifies attention sinks and residual sinks as outlier-driven rescaling; replaces the implicit mechanism with explicit Gated Attention / GatedNorm, yielding pretraining gains and W4A4 quantization robustness. Locates a previously-unexamined coupling between training stability and low-precision friendliness.
  • [2026-05-22] How to Set the Learning Rate for Large-Scale Pre-training?: shows µP’s motivating attention-logit instability is suppressed by QK-Norm under SP, so module-specific LRs no longer pay off and the Fitting paradigm (scaling-law extrapolation of LR*(N,D)) beats µTransfer at 4B and 12B MoE scale.
  • [2026-05-22] mHC: Manifold-Constrained Hyper-Connections: makes residual-stream geometry the load-bearing stability primitive by constraining the residual-mixing matrix to the Birkhoff polytope, suppressing 27B-scale loss spikes and bounding composite signal gain at any depth.

Open questions

Section titled “Open questions”
  • All three architectural-stability papers locate stability in different places in the network — Q/K projections (QK-Norm), attention output + RMSNorm (GatedNorm / Gated Attention), residual mixing (mHC). Are these complementary (stack them) or do they substitute for one another at sufficient strength? No filed paper has run the stacked ablation.
  • Architecture-side vs optimizer-side stability fixes are now an explicit fork. SSO (Controlled LLM Training on Spectral Sphere) suppresses outliers and bounds activations as a side-effect of spectral retraction; Gated Attention (A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training) preserves outliers but adds explicit learned rescaling; mHC bounds composite signal gain at the residual-mixing level. Do these stack (SSO + Gated Attention + mHC) or do they each saturate the same underlying degree of freedom (bounded effective spectra of the forward/backward map)? No filed paper has done the cross-comparison.
  • Pretraining vs finetuning may be a real split. How to Set the Learning Rate for Large-Scale Pre-training? declares µP’s stability motivation stale for from-scratch pretraining under QK-Norm; Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning shows µP-style analysis remains predictive for LoRA finetuning. Whether this means “stability is solved by architecture, but rank/init/multiplier coupling needs separate parametrization analysis” or “µP’s relevance is regime-dependent in a way no filed paper has fully mapped” is open.
  • The Gated Attention story explicitly couples stability with quantization-robustness — outliers are both why training works and why PTQ fails. Does the same coupling apply to mHC’s residual-stream constraint (does bounding composite gain to ~1.6× also make activations PTQ-friendly)? SSO claims the same coupling (bounded activations → low-precision friendly). The mHC paper does not measure this; the SSO paper asserts it without quantifying PTQ deltas.
  • µP was designed for LLM decoder-only stacks. Do these architecture-first stability fixes transfer to DiT / MMDiT pretraining, where AdaLN modulations are an additional outlier source and the residual streams carry conditioning rather than just token state? Untested across all five filed papers — though ViT-5: Vision Transformers for The Mid-2020s supplies a partial answer for plain SiT backbones (QK-Norm transfers cleanly to vision; SwiGLU does not) and Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning does run Stable Diffusion finetuning (Init[B] + α=1\alpha=1 optimum holds; FFT is sharply LR-sensitive on UNets). The SSO Megatron fork is the lowest-friction way to test the optimizer-side hypothesis on a DiT codebase.
  • Cross-modality component transfer is not uniform: ViT-5 finds QK-Norm, RMSNorm, LayerScale, RoPE, and bias-free-QKV all transfer from LLMs to ViTs, but SwiGLU does not (over-gates with LayerScale). Is there a similar transfer asymmetry going the other direction — components that work in ViTs but not LLMs? The fact that LayerScale ≈ post-RMSNorm in ViTs (ViT-5: Vision Transformers for The Mid-2020s §3.1, Table 1) but LayerScale is rare in LLMs raises this question directly.
  • The Fitting paradigm in How to Set the Learning Rate for Large-Scale Pre-training? fits LR*(N, D) under WSD. If the architectural stability story is right, the LR* surface should become flatter under stronger normalization (less penalty for slight LR overshoot). Could be a clean prediction to falsify. SSO’s claim that µP transfer still holds under spectral-sphere constraints is a related, testable prediction.
  • All papers run under LLM-style pretraining (decoder-only, large-N MoE). Do the same instabilities (sinks, identity-violation, attention-logit blowup, 1/3-exponent slope) appear at the smaller-N high-batch regime that diffusion-from-scratch pretraining actually lives in? The 1/3 paper makes a specific prediction here: MSE / flow-matching objectives should not exhibit the softmax-CE bottleneck and so the DiT slope should be in a different universality class.
  • If the 1/3 exponent is structural to softmax-CE on peaked targets, does it survive when the target distribution is broadened (label smoothing, mixture-of-token-distributions distillation, temperature-scaled teachers)? A clean falsifier for whether the slope can be moved.
  • Is “Muon is half-aligned with µP” the right framing, or does Muon’s implicit weight-spectrum control via update orthogonalization already deliver most of SSO’s gains? SSO’s MuonSphere variant ablates exactly this — claim is the gap matters, but the precise delta between Muon and MuonSphere is the load-bearing experiment for this whole line.
  • Does QB’s LP framing generalize to other discrete-routing decisions beyond MoE expert assignment — e.g. mixture-of-depths gating, conditional-compute heads, layer-skip routing? QB’s claim is that the routing target (uniform load) is what makes the LP solvable in closed-quantile form; whether other targets (e.g. accuracy-weighted load, capacity-aware load) admit similar one-shot solutions is open.
  • How does QB interact with the SSO finding that spectral-sphere training “improves MoE router load-balancing” as a side-effect? If the optimizer already bounds router activations and reduces outlier-driven imbalance, do you still benefit from QB on top, or do they substitute? Both touch MoE balancing but from different loci (optimizer vs. routing-time correction).
  • What does framework-default Muon (no distributed support) imply for adoption? Per Addition of Muon optimizer to torch.optim — PyTorch team welcomes PR (Issue #148819), torch.optim.Muon shipped without FSDP/HSDP support, and Kimi K2’s deployment explicitly disables FP8 GEMM for the Newton–Schulz step. The follow-on optimizer papers (SSO, Aurora, Newton-Muon) increasingly target the implementation gap — distribution, FP8, leverage-awareness — rather than the algorithm itself. Whether one of these becomes the canonical “distributed Muon” recipe in torchtitan is the live infra question.

Papers

Section titled “Papers”

2026-05

Section titled “2026-05”

2026-04

Section titled “2026-04”

2026-03

Section titled “2026-03”

2026-02

Section titled “2026-02”

2026-01

Section titled “2026-01”

2025-12

Section titled “2025-12”

2025-11

Section titled “2025-11”

2025-10

Section titled “2025-10”

2025-09

Section titled “2025-09”

2025-08

Section titled “2025-08”

2025-07

Section titled “2025-07”

2025-06

Section titled “2025-06”

2025-05

Section titled “2025-05”

2025-04

Section titled “2025-04”

2025-03

Section titled “2025-03”

2025-02

Section titled “2025-02”

2025-01

Section titled “2025-01”

2024-01

Section titled “2024-01”

2020-12

Section titled “2020-12”

Unknown date

Section titled “Unknown date”