Skip to content
Bud's Wiki

Training stability at scale

TL;DR

Section titled “TL;DR”

A late-2025/early-2026 cluster of papers locates large-model training stability in architectural choices (normalization variants, residual-mixing constraints, gated rescaling) rather than in parametrization recipes like µP. The shared empirical claim is that modern normalization (QK-Norm, RMSNorm, GatedNorm) and constrained residual geometry (Birkhoff-polytope residual mixing) subsume what classical parametrization tricks were originally designed to fix — and in some cases the architectural fix is strictly better because it also removes pathologies (outlier activations, identity-mapping violations across depth) that the parametrization fix never addressed.

Key claims

Section titled “Key claims”
  • Under modern architectures, the attention-logit blow-up that µP was designed to suppress simply does not occur under standard parametrization; ablating QK-Norm restores the instability µP reports, so the stability is doing the work, not the parametrization (How to Set the Learning Rate for Large-Scale Pre-training? §6.3, Figs. 8–9).
    • Conflict: Controlled LLM Training on Spectral Sphere argues the opposite resolution — µP’s motivation still stands, but current optimizers (including Muon) are only “half-aligned” (they control update spectra but allow weight spectra to drift), and the right fix is to constrain both weights and updates to a spectral sphere rather than rely on QK-Norm to mask the symptom (Controlled LLM Training on Spectral Sphere README §1, Abstract). The two papers prescribe directly opposing actions: drop µP and lean on normalization (2601.05049) vs. enforce µP harder via the optimizer (2601.08393).
    • Finetuning-side replication of the parametrization position: Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning shows that for LoRA finetuning, µP-style asymptotic analysis (extended to the joint n,r→∞n, r \to \infty limit as µA) still yields tight, predictive scaling rules for the optimal LR — including a configuration (Init[B], α=1\alpha=1) where LoRA’s optimal LR transfers directly to FFT (Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning §4.2, Fig. 5). The µP framework is not dead in the finetuning regime even if its motivating instability has been suppressed in from-scratch pretraining.
    • Vision-side replication: ViT-5: Vision Transformers for The Mid-2020s reports that adding QK-Norm to ViTs gives a modest accuracy bump and a large reduction in sharp loss spikes during optimization — the same spike-suppression role QK-Norm plays in LLMs (ViT-5: Vision Transformers for The Mid-2020s §3.6, Fig. 5).
  • Module-specific learning rates (a µP prescription) yield essentially no measurable improvement over a single global LR at 4B-MoE scale under modern normalization, because per-module AdamW update magnitudes are already uniform across layers (≈0.2) consistent with rotational equilibrium (How to Set the Learning Rate for Large-Scale Pre-training? §6.1–6.2, Figs. 5–7).
  • Attention sinks and residual sinks both arise from the same functional pressure — outliers paired with softmax/RMSNorm act as implicit rescale factors for non-outlier components, which is load-bearing for training stability (removing normalization or clipping outliers both degrade training) (A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training §3, Abstract conclusions (1)–(2)).
  • The implicit rescaling can be replaced with explicit learned gating (Gated Attention, GatedNorm), giving ≈+2-point average pretraining gain and ≈1.2-point less W4A4 quantization degradation — i.e. architecture choices that maintain stability and produce low-precision-friendly activation distributions (A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training Abstract conclusion (3), Table 1).
  • Unconstrained Hyper-Connections break the identity-mapping property of residual stacks at scale, producing forward/backward signal gains up to ~3000× and visible 27B-scale loss spikes; constraining the residual mixing matrix to doubly stochastic matrices (Birkhoff polytope) via a differentiable Sinkhorn projection bounds composite gain to ~1.6× across arbitrary depth (mHC: Manifold-Constrained Hyper-Connections §3.1, §4.1, §5.4, Figs. 2/3/7).
  • Architectural stability fixes hold across compute and data scaling — the loss-improvement gap of mHC over baseline is maintained from 3B → 9B → 27B, and across a 1T-token run at 3B (mHC: Manifold-Constrained Hyper-Connections §5.3, Fig. 6).
  • The slope of LM pre-training scaling laws — not just the intercept — appears to be set by a structural primitive (softmax + cross-entropy on peaked target distributions), yielding a universal 1/3 time-scaling exponent that interventions like optimizer choice, data mixes, and most architecture tweaks do not change (Universal One-third Time Scaling in Learning Peaked Distributions Abstract, Figs. 1–3). This is the “tyranny of the slope” framing: stability/architecture interventions move the intercept; only loss/output-head changes can move the slope.
  • Constraining both weight matrices and updates to a per-module spectral sphere of radius R=Θ(dout/din)R = \Theta(\sqrt{d_\text{out}/d_\text{in}}) — the steepest-descent direction under the spectral norm, with hard or dynamic retraction — outperforms AdamW and Muon on Dense 1.7B, MoE 8B-A1B, and a 200-layer DeepNet while suppressing outliers, bounding activations, and improving MoE router load-balancing (Controlled LLM Training on Spectral Sphere README §1, §2, §4). This is the optimizer-side analogue of the architecture-side mHC residual-mixing constraint: same manifold-constraint recipe, applied to a different stability locus.
  • Architectural stability components do not transfer trivially across modalities: porting SwiGLU MLPs (a standard LLM choice) into ViTs alongside LayerScale produces over-gating — both act as channel-wise filters, the combination drives intermediate representations to excessive sparsity, and accuracy drops (84.16% → 83.70% at ViT-B, FID 14.57 → 16.22) (ViT-5: Vision Transformers for The Mid-2020s §3.3, Table 2). The QK-Norm, RMSNorm, LayerScale, and bias-free-QKV pieces do transfer; SwiGLU does not.
  • MoE router stability has a parallel “remove the tuning axis” story: Quantile Balancing (QB) reformulates per-step expert assignment as the linear program “find the bias vector that would have produced balanced load last step” and solves it via alternating quantile iteration — eliminating the aux-loss coefficient, the fixed-bias-update magnitude, and the momentum coefficient that prior recipes (GShard, DeepSeek loss-free, Arcee SMEBU) all required. Validated at 32B-A5B / 1e22 FLOPs / 326B tokens on Marin with zero loss spikes and no leading dense layers (Quantile Balancing: A Hyperparameter-Free MoE Load Balancing Method).

Recent contributions

Section titled “Recent contributions”
  • [2026-11-06] Rethinking Activation Functions: Reducing Bias Can Improve Learning: Peng, Zheng & Liu (Tsinghua / MetaCircle) name activation bias — the nonzero post-activation mean of ReLU/GELU/Swish/Sigmoid — as a new, upstream stability locus distinct from the architecture / optimizer / parametrization / precision / loss-implementation loci already on this page. In a controlled residual-MLP teacher–student, subtracting the activation’s expected post-activation output every layer slows early learning but overtakes the unshifted baseline on final loss, drives top Hessian eigenvalues substantially smaller throughout training, and separates the two students’ parameter-update directions into nearly-orthogonal subspaces. An adaptive-shifting variant (subtract per-step per-layer across-neuron mean) scales the fix to ViT-B / DiT-B/2 / GPT-2 Small and yields small but consistent late-stage improvements across image classification, image generation, and language modeling, with the hidden-state overshoot signature from the controlled model recognizably reproduced on ViT-B and markedly suppressed under adaptive shifting. Sibling in mechanism to A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training (outlier-driven implicit rescaling via softmax/RMSNorm) and mHC: Manifold-Constrained Hyper-Connections (residual-mixing manifold constraint) but relocated one step upstream — in the activation function itself, zero added parameters. Yann LeCun’s quote-reply pointing back to 1990 NeurIPS work is on record: the pathology is classical, the frontier-scale measurement is new.
  • [2026-11-06] Introducing Inkling 2 Small — Thinking Machines' 594B/17B MoE with unified RL at 4e23 FLOPs: Inkling 2 Small is the first filed NVFP4-native pretraining run at 594B-total scale: H1’s 8.4T tokens run natively in NVFP4 on NVIDIA GB300 NVL72 racks in 68 hours. NVFP4 is retained for both training and sampling through the subsequent 9-day joint RL run; NVFP4 KV cache variant for inference “minimally lowers evals”. Resolves The 4-bitter Lesson: Balancing Stability and Performance in NVFP4 RL‘s three-way FP16 / BF16 / selective-NVFP4 debate on the quantized-recipe side at this scale — but defers the specific NVFP4 recipe (whether dequantized-backward, Four-Over-Six, per-token-FP32 activation scales, or selective BF16 layers are load-bearing) to a follow-up post. Carries the v1 Inkling stability primitives (hybrid Muon + Adam, LR²-coupled weight decay, relative positional bias, short convolutions on residual branches, 5:1 SWA:GA) unchanged.
  • [2026-11-05] The Ultimate MoE Training Handbook: training Mixture-of-Experts models with Megatron-LM, setting by setting: Recipe-side guide whose Chapter 3 names router + load balancing + expert block layout as the three MoE-specific stability axes, with recommended settings referencing recent open MoE releases. Lives at the practitioner layer below the individual stability-locus papers tracked here (Demons-in-the-Detail LBL scope, Quantile Balancing, silent expert death, ID Balancing).
  • [2026-11-05] ERNIE 5.5 Flash Base: ERNIE-5.5-Flash-Base deploys Multi-Router Latent MoE as an architecture-side fix for the silent-expert-death / router-self-reinforcement pathology that Mitigate Silent Expert Death In Ultra-Sparse MoE diagnoses and ID Balancing: Stable Training of Extremely Sparse MoE via PID-Based Load Control controls with PID balancing: several independently-initialized sigmoid routers jointly score experts so that a wrong top-k choice no longer leaves the un-chosen experts gradient-starved — the signal to correct comes from the routers that did consider them. Pairs with Stable LatentMoE (RMSNorm before up-projection + SiTU-GLU) and per-expert learnable routing scalars to keep expert-output scale uniform across 43 layers. On the distributed-Muon systems axis, moves dense-parameter all-reduce onto the same expert-parallel ring MoE uses and combines dense + MoE sync into one collective (larger messages, lower inter-node jitter), plus inter-node balanced sharding that keeps optimizer-state memory flat across nodes — a production answer to the “framework-default Muon lacks FSDP/HSDP support” open question and the memory-concentration failure mode this page’s open questions name. Trained 33.5T tokens FP8 with context extended to 128K during pretraining.
  • [2026-11-03] Pretraining Data Scale Reverses the Muon-AdamW Fine-Tuning Gap: Li, Gai, Risteski, Springer & Raghunathan (CMU) report a sign-flip in the Muon-vs-AdamW ranking as pretraining data budget grows: Muon wins at small-to-moderate TPP but AdamW overtakes Muon after fine-tuning beyond a sufficiently large data budget, with the crossover TPP decreasing as model size grows. Proposes a margin-to-spread ratio mechanism — distance from correct logit to next-wrong logit divided by wrong-logit spread — that tracks the crossover in both LMs and a provable linear-associative-memory toy model. First filed paper arguing Muon’s advantage inverts at scale rather than narrows, complementing The Best Optimizer Depends on Batch Size‘s batch-size-axis reordering and To Use or not to Use Muon: How Simplicity Bias in Optimizers Matters‘s inductive-bias critique. Directly sharpens Kyle’s team-internal observations that tuned AdamW beats tuned Muon on small LMs at ~100 TPP and on 500M T2I diffusion at compute optimality, and the “no one has done a real at-scale AdamW vs Muon ablation” critique circulating since Kimi K2.
  • [2026-11-02] Compute Better Spent: Replacing Dense Layers with Structured Matrices: Qiu et al. (2024) report that training small GPT-2 on OpenWebText with BTT (Block Tensor-Train) structured layers suffers slow unbounded growth of pre-LN activations that eventually diverges training — a stability failure mode absent in the dense baseline. Fix: a weight-normalization reparameterization that clamps each BTT core’s RMS to its initialization scale and reintroduces magnitude via a learnable scalar (Salimans & Kingma 2016 style), eliminating the activation drift without sacrificing expressivity (Fig. 7). Adds a structured-matrix instance to the architecture-first stability pattern tracked here — the µP-correct LR schedule alone is not sufficient; the parameterization also needs an explicit norm constraint.
  • [2026-10-26] Full-Weight VLA Fine-Tuning Without an fp32 Master Copy Silently Erases Most Updates: Singh extends the low-precision-stability-locus thread (Defeating the Training-Inference Mismatch via FP16, The 4-bitter Lesson: Balancing Stability and Performance in NVFP4 RL, A Fix for Softmax and Cross-Entropy) from LLM pretraining and RL post-training into VLA fine-tuning: with trainable weights in bf16 and no fp32 master copy, AdamW rounds any update < w/512 to zero, probe measures ≥79% of per-step updates silently erased. 9/12 open VLA codebases default-broken (SmolVLA, π₀/π₀.₅ guides). One-line fp32-master-copy fix lifts SmolVLA 4.3→49.3% and π₀.₅ 89.0→97.7% on LIBERO-Object with p<0.05 across 7 controlled comparisons. First filed instance of the “silently-erased optimizer update” failure mode at a fine-tuning scale rather than pretraining — adjacent in mechanism to Laguna M.1/XS.2 Technical Report‘s FP32-RMSNorm + FP32-master-weights pretraining defaults but applied to the ignored short-horizon post-training regime.
  • [2026-10-25] Andrew Carr — SuperMimic open reimplementations diverge during Bones Seed; four engineers compare notes: Andrew Carr (Cartwheel) reports that his team’s and other open reimplementations of SuperMimic (SuperMimic: Scaling Humanoid Whole-Body Control with One Differentiable World Model) all diverge ~20% through “Bones Seed” training. Thread surfaces three concrete stability-loci replies for the differentiable-world-model + BPTT recipe: Lizhi Yang’s morphmimic finds the critic overestimates without SHAC-style targets (LayerNorm critic only postpones divergence to ~80k iters; clamping TD targets at r_max/(1−γ) stops it on resume); Kyle Morgenstein notes rewards must be differentiable for BPTT and that regularization is especially load-bearing for hardware transfer; Sammo reports a stable port after 11 failed runs (0→68k BONES-SEED + 18k fine-tune to 86k) but still short of the paper’s 452-clip failure count and has not run the full 320k schedule. First filed instance of the architecture-first stability discipline being needed for a model-based RL analytic-gradient inner loop rather than LLM pretraining.
  • [2026-10-25] CroissAttn — fixing FlashAttention backward's zero-sum bug for end-to-end FP4 training: Alexander Johansen (Stanford) announces CroissAttn (ICLR’27 submission), the FP4-stochastic-rounding edition of the Barron/Gnanapragasam/Tschannen zero-sum fix (A Fix for Softmax and Cross-Entropy): the standard FA backward’s δ term is computed from the stored high-precision O, but under stochastic FP4 rounding dP is a random draw and the shortcut is wrong on every draw. One-line fix (compute δ from the rounded dP row, matched mean + exact zero row sum per draw) concentrates on attention-sink rows, where the sink key’s true gradient is near zero so O’s rounding error dominates — the shortcut’s error scales from ~1× (diffuse) to 2–3× (peaky) to unbounded on a sink. 150 draws on one sink row: shortcut inside 1× of the true gradient 1/150, CroissAttn 150/150. Also notes an RL cousin where FP4 rollouts amplify Petrenko §4.1’s BF16 importance-ratio / entropy-collapse pathology by ~100×, suggesting exact-replay-style correction may compose. Explicitly inherits Barron’s framing and extends it from the LM head to the attention backward — the loss-implementation-locus and the training-side-low-precision-locus overlap here.
  • [2026-10-23] A Fix for Softmax and Cross-Entropy: Barron, Gnanapragasam & Tschannen (Google DeepMind) add a loss-implementation locus distinct from the architecture / optimizer / parametrization / precision loci this page tracks: standard softmax / log-softmax / cross-entropy in JAX, PyTorch, and textbooks have two numerical bugs — Softmax Collapse (target-logit gradient rounds to zero at high confidence, so gradients no longer sum to zero) and Softmax Cancellation (non-zero gradients between equal student/teacher logits during distillation warm-start). A drop-in rewrite fixes both. Empirically eliminates Slingshot spikes, grokking proceeds smoothly past the fp32 cliff at m ≈ c₃₂ ≈ 16.6 and saturates at m ≈ 18.4 across all five seeds, prevents accuracy collapse without weight decay, keeps distilled students pinned to the teacher at init, and shows post-memorization logit growth is bounded by an equilibrium between weight decay and Adam’s ε — making mechanistic sense of the folklore that default eps=1e-8 sits “just past” the Collapse threshold in fp32. Bug was originally found by Gemini Flash 3.7; most experiments by Flash 3.8; first draft by Gemini 4 Argon — a frontier model identifying a bug in a universally deployed primitive.
  • [2026-10-23] Lucas Beyer — curb the enthusiasm on zeroth-order LM training; loss 4.2 is near the unigram floor: Lucas Beyer posts the first filed outside-cluster skeptical read on the Noise / Dust / spicylemonade zeroth-order pretraining wave — two concrete objections: (1) genetic-algo and forward-diff methods are known to fall off as “intrinsic” parameter space grows (the entire motivation for backprop’s few-outputs-many-params asymmetry), and (2) validation loss 4.2 on a 30M-param LM is near the token-unigram floor so “matches backprop” at that loss does not demonstrate difficult learning yet. Belongs on the record as the “credit assignment is still the load-bearing stability locus” counter to the Dust lineage.
  • [2026-10-23] Noise: Speedrun for pretraining without backpropagation: Q Labs opens the Noise benchmark — a public pretraining-without-backprop speedrun with Dust as baseline — and reports a >4000× Track 2 compression in ten days (1.76 h → 1.6 s), with Track 1 passing backprop + Adam (3.769) but still trailing backprop + AdamW; the single jump cited for both records is @spicylemonade’s full-Hadamard Boost → Cut substitution, making this the first filed example of credit-assignment-as-stability-locus being searched by an open leaderboard rather than one lab.
  • [2026-10-22] OPAL: a reactive optimizer with an agent in the loop: OPAL (Bai, Liang, Teoh, Sitzmann; MIT) adds a reactive, per-run stability locus distinct from the architecture-first / optimizer-first / parametrization-first / precision-first positions already on this page: a coding agent reads training dynamics every ~50 steps and rewrites the optimizer program itself when it sees pathologies, with documented emergent behaviors of detecting loss spikes and zeroing them out, resetting momentum and rolling back after overshoots, and designing LR schedules mid-run. Positioned by the authors as the end of the “pick optimizer + LR schedule before the run and never touch them” default — raising the question of whether a reactive agent substitutes for or composes with the construction-level stability recipes (QK-Norm, Gated Attention, mHC, SSO, Hyperball) that dominate this page. No ablation of which agent decisions are load-bearing is reported.
  • [2026-10-21] SuperMimic: Scaling Humanoid Whole-Body Control with One Differentiable World Model: SuperMimic (Berkeley; Liao et al.) reports the sharpest filed compute-reduction datapoint for humanoid motion tracking — >1000× GPU-hour reduction (3 h / 1 GPU vs 7 days / 128 GPUs for a SOTA model-free motion-tracker baseline) via a differentiable world model + BPTT analytic policy gradients instead of model-free PPO. Load-bearing stability claim: flat world-model prediction error from 1 h to 264 h of motion data, with GPU-hours-to-target-return growing only slowly over the same 2.4-order-of-magnitude range — a measured scaling signature that argues the regime is compute-bound rather than optimization-bound in this setting.
  • [2026-10-20] spicylemonade — 2.3× pretraining speedup without backprop, inspired by Cammarata's 'Neural Networks Are Already Clean': spicylemonade reports a 2.3× speedup on Dust-style activation-space zeroth-order pretraining by selectively skipping credit assignment for neurons the Cammarata “already clean” interpretability frame identifies as useless — the first filed evidence of interpretability-derived priors cashing in as training-time compute savings in a non-backprop regime.
  • [2026-10-19] Dust: Pretraining Transformers Without Backpropagation: Dust (Quest Labs; Dahal, Mandal, Gülbahar, Vegesna) proposes a new structural-primitive locus distinct from architecture/optimizer/parametrization/precision: credit assignment itself. Replaces backprop with activation-space node perturbation where each token is a “virtual population” member evaluated in one forward pass — the token axis becomes orthogonal to EGGROLL’s batch-axis population. At FineWeb pretraining up to 20M tokens / 243M params it matches or exceeds backprop at large population, is 10³–10⁴× more population-efficient than EGGROLL, and — inverting the textbook ES story — bigger models are more population-efficient (120× parameter sweep). Backprop-cosine fits saturate per layer-type and stay flat across 10M→1B tokens, hinting the required-population floor does not grow with tokens. The “architecture no longer needs to be end-to-end differentiable” implication opens non-differentiable looped/program-in-the-loop stacks that the current stability cluster has no answer for.
  • [2026-10-15] Frontier Pre-training at Microsoft AI (MAI-Base-2): MAI-Base-2 reports activation gradient clipping as a stability lever at 2.8T-MoE / 16K-GB200 scale: rather than scaling the entire batch’s gradient norm when a small portion drives an outlier, activations at selected network locations are clipped locally, keeping gradient norm in a much tighter range than MAI-Base-1. Reported EG gain grows with scale (Fig. 8) — unusual on this page where most changes give a roughly constant per-change benefit, suggesting activation gradient clipping compounds with the regime where outlier-token-driven batch-level clipping would otherwise lose increasing amounts of useful gradient. Doubles as a data-validation diagnostic: a clipped example was traced back to a malformed training example (license header cut off mid-sentence) that revealed a preprocessing-pipeline bug.
  • [2026-10-08] The Best Optimizer Depends on Batch Size: Evidence that Muon lacks a principled scaling rule that holds across batch sizes — the best optimizer identity itself changes with batch size after per-setting tuning. Frames batch-size robustness as a stability-at-scale axis orthogonal to the geometric fixes (weight-sphere / spectral-sphere retraction) tracked elsewhere on the page.
  • [2026-10-01] A Mathematical Framework for Transformer Circuits: Backfills the 2021 Anthropic residual-stream framing — bus with ~100× bandwidth pressure, communication in superposition across subspaces, memory-management neurons that negate prior writes — that the modern “small architectural surgery on the residual stream” cluster (Attention Residuals (AttnRes / Block AttnRes) — Technical Report, Multi-Head Attention Residuals, mHC: Manifold-Constrained Hyper-Connections, Mean Mode Screaming: Mean-Variance Split Residuals for 1000-Layer Diffusion Transformers, A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training) is implicitly operating against. Akshat (Slack) notes “all these residual stream hacks are basically just this.”
  • [2026-10-01] ID Balancing: Stable Training of Extremely Sparse MoE via PID-Based Load Control: Reframes two production MoE load balancers as PID controllers (DeepSeek loss-free = integral, Kimi K3 Quantile Balancing = proportional) and proposes ID Balancing — an Integral-Derivative controller — as a router-side stability intervention. At 18.9B → 69.9B / Top-10-of-768, worst-case backbone MaxVio stays nearly constant under ID Balancing and sits ~89.6% below the auxiliary-loss baseline, with gains widening as sparsity deepens. Strengthens the “router as a named stability locus” argument from Mitigate Silent Expert Death In Ultra-Sparse MoE and Quantile Balancing: A Hyperparameter-Free MoE Load Balancing Method by providing a control-theoretic parametric family rather than isolated recipes.
  • [2026-09-30] Generalization Dynamics of LM Pre-training: Wen, Wu, Song & Chen introduce a new dynamics-side stability locus distinct from the architecture / optimizer / parametrization / precision loci this page tracks: mode-hopping between parrot-like (memorization, in-context pattern match, System-1, truthiness) and intelligence-like (in-context learning, System-2, multi-hop, truthfulness) computations across pretraining, measured by a toy eval suite of matched task pairs run densely along the training trajectory. Mode-hops are locally stable and not fixable by checkpoint averaging — ruling out the standard “high-frequency optimization noise” explanation and joining Cracks in the Foundation: Seemingly Minor Architectural Choices Impact Long Context Extension in the set of stability failure modes that are invisible to loss trajectories and validation-set aggregates. The proposed capacity-allocation account (shallow circuits learned early compete with generalizable circuits for bounded capacity; each pretraining-data window can tip the balance) is a representation-level companion to the numerical-instability accounts on this page. Practical consequence for the “ship the final checkpoint” default baked into most scaling-law work (Scaling Laws for Neural Language Models, How to Set the Learning Rate for Large-Scale Pre-training?): selecting an intermediate checkpoint via the suite yields strictly better reasoning and alignment generalization than the final pre- or mid-training checkpoint, and using the suite as a data-selection signal reduces mode-hop severity.
  • [2026-09-30] EasyPPO: Stabilizing the Critic Is Key: EasyPPO (Zhou et al., UC Berkeley + Princeton) identifies heterogeneous return variance across prompts as a finite-batch instability mode for the PPO critic in LLM RL: a small subset of high-variance prompts contributes disproportionately large gradients, and the fix is a one-line reweighting of each prompt’s critic loss by the inverse standard deviation of its sampled returns. Empirical plot shows critic-gradient-norm-vs-prompt-return-std goes from strong positive dependence to near-flat after normalization. Companion diagnoses: filtering truncated rollouts from both actor and critic shifts the policy objective; smaller critic mini-batches limit outlier reach through gradient clipping. Together these three orthogonal changes are the only combination among vanilla PPO / actor-only-filtered PPO / HL-Gauss PPO / VAPO that stays stable across three settings (FrontierCS / AIME24 / Search-R1) at rollout batches 512–1024 on Qwen3.5-9B — a critic-side companion to the actor-side (Score Centering Stabilizes Off-policy Reinforcement Learning) and precision-side (Defeating the Training-Inference Mismatch via FP16) accounts of LLM-RL instability already on this page.
  • [2026-09-23] Scaling Diffusion Policy in Transformer to 1 Billion Parameters for Robotic Manipulation: ScaleDP (Sept 2024) is the earliest filed instance of the “architecture-first” stability story applied to visuomotor policies: diagnoses that vanilla Diffusion Policy in Transformer (DP-T) fails to scale beyond ~8 layers because cross-attention observation fusion — not attention logits or the residual stream — grows per-layer gradient-magnitude standard deviation with depth, degrading task success (MetaWorld: 8L 80.1% → 12L 78.4% → 14L 74.6%). The fix is a direct DiT-style AdaLN replacement of cross-attention conditioning (γ, β regressed per-block from summed timestep + observation embeddings), which normalizes parameter distribution and keeps gradient magnitudes low up to 32L × 1280d × 1B params. Sibling in mechanism to CMuon: Accelerating and Stabilizing Diffusion Transformer Training via Chunked Momentum Orthogonalization (CMuon isolates a different DiT-specific fusion pathology at the AdaLN + QKV fused-tensor boundary in the optimizer, not the training objective) and predates by ~18 months the A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training “outlier-driven implicit rescaling” framing on the LLM side.
  • [2026-09-22] MiMo-V2.6: Scaling Reinforcement Learning Towards Self-Improvement: MiMo-V2.6 introduces Muown — a Muon variant with explicit row-norm control that mitigates spectral-norm drift and reduces weight-decay sensitivity — and switches an AdamW-pretrained 1T MoE to Muown mid-training on hidden weights (embeddings/LM head/router stay on AdamW) with no loss spike, contra the prior warning from Qu et al. 2026 that Adam→Muon mid-run causes optimizer mismatch. Enabled concurrently with MXFP4 QAT. Also names router freezing as an RL-time stability primitive at scaled ultra-sparse MoE, orthogonal to the FP16/BF16/NVFP4 precision debate.
  • [2026-09-19] Score Centering Stabilizes Off-policy Reinforcement Learning: Score Centering (Marek & Ryabinin) locates the training-inference-mismatch instability in an algorithmic locus distinct from the precision-side (Defeating the Training-Inference Mismatch via FP16, The 4-bitter Lesson: Balancing Stability and Performance in NVFP4 RL) and variance-correction (TIS/MIS) framings tracked on this page: a covariance decomposition of the policy gradient into a reward-correlated signal term and a reward-independent drift term, with drift the load-bearing accumulator under off-policy sampling. Additive one-term correction (subtract expected trainer score under sampler) cancels drift, composes with importance sampling, and matches/beats TIS/MIS across 0.6B–30B under quantization. Sits alongside FP16 and NVFP4 as a fourth-flavor stability lever for the RL post-training stage — bias-side rather than variance-side or precision-side.
  • [2026-09-17] Training Text-to-Image Models 3.6× Faster (JiT-DDT): Linum’s JiT-DDT recipe combines Muon on all 2D weight matrices (Q/K/V, FFN) with AdamW on input/output layers, scales, and AdaLN biases — matching the Kimi-style hybrid pattern — plus per-head sigmoid attention gating (23 gates × N heads per block) to prevent attention sinks in the single-stream DiT. Noise-schedule widening was reported as the single load-bearing training change over vanilla JiT.
  • [2026-09-13] Scaling Laws for Neural Language Models: Foundational back-fill of the original Kaplan scaling laws paper — establishes that Transformer LM training loss curves are predictable from smooth power laws in (N,D,Cmin)(N, D, C_\mathrm{min}), and that the critical batch size Bcrit(L)B_\mathrm{crit}(L) is a function of loss alone. The methodology behind isoFLOP fits, batch-size-adjusted step counts SminS_\mathrm{min}, and compute-optimal frontier projection that the entire modern “Fitting” paradigm (How to Set the Learning Rate for Large-Scale Pre-training?, Delphi, Marin) inherits.
  • [2026-09-13] An Empirical Model of Large-Batch Training: Foundational back-fill: the [MKAT18] paper that predicts the optimal LR schedule η∗(B)=ηmax/(1+Bnoise/B)\eta^*(B) = \eta_\mathrm{max}/(1 + B_\mathrm{noise}/B) from the gradient noise scale for SGD+momentum, and gives an empirical Adam/RMSProp counterpart (power-law-then-plateau). Precedes and grounds the rotational-equilibrium / spectral-sphere / AdamH line of work that argues LR stability is really about controlling update magnitudes rather than raw learning rate.
  • [2026-09-12] Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data: Overfitting-under-repetition as a new stability failure mode specific to MoE: matched-active-param MoEs degrade faster than dense (start suffering at 4× vs 8× for dense) and lose their dense-baseline advantage past 32× repeats; masking regularization (strong dropout) partially rescues them but never matches all-unique-data.
  • [2026-09-08] >10x More Efficient Pretraining: Magic’s public recipe frames “stable foundation first — smooth convergence, low-precision matching FP32, stable infra, correct HP scaling — then hunt bugs” as the prerequisite for compounding tens of small changes into a >10× compute-efficiency win. Two explicit anti-transfer claims: NanoGPT-speedrun wins often fail at scale, and some features present in most LLMs can be deleted without harming large-scale performance — sharpening the wiki’s “small-model surface has diverged from frontier surface” thread and reinforcing How to Set the Learning Rate for Large-Scale Pre-training?‘s finding that µP-motivated interventions stop paying off under modern normalization. Bug fixes explicitly framed as compute multipliers.
  • [2026-08-31] Sliding-window beats linear attention: Practical corollary of the “sinks are load-bearing” thesis (A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training, StreamingLLM and the Discovery of Attention Sinks): permanently-retained attention sinks under a sliding-window mask are enough to preserve long-context behavior of a pretrained LLM at inference without any post-training, and beat linear-attention retrofits by 2–10× on NIAH/BABILong. Sink-preservation as an inference-time knob, not just a training-time architectural choice.
  • [2026-08-28] OpenDM — DM0.5: An Open-World Foundation Model for General-Purpose Embodied Intelligence: DM0.5 (Dexmal) uses separate learning-rate groups for the VLM backbone and the Action Expert — VLM backbone trained at a smaller LR to reduce catastrophic forgetting and preserve general vision-language capabilities during action co-training. Simpler-than-Anchor-Align answer (a knob rather than a distillation loss) to the same “grounding wall” failure mode Pretrained to Imagine, Fine-Tuned to Act: The Rise of World-Action Models VLM2VLA and Anchor-Align: Generalizable VLA Finetuning via Representation Anchoring and Language-Action Alignment Anchor-Align target. Combined with an internal automatic VL data generation pipeline (image + video + visual instruction) so the VLM branch is both slowed and fed during co-training rather than only slowed.
  • [2026-08-26] Qwen3.8-Flash-Next — Hybrid Attention with QSA, Gated Residual, N-gram Embedding: Qwen3.8-Flash-Next introduces Gated Residual — 4-branch widened residual streams with rank-320 bottlenecks, modulated by an element-wise data-dependent read gate and per-branch scalar write gate. Positioned as a residual-stream surgery in the AttnRes / mHC / QK-Norm family: preserves training stability while adding finer-grained cross-layer expressiveness, with inference overhead kept low.
  • [2026-08-25] Mitigate Silent Expert Death In Ultra-Sparse MoE: Adds an auxiliary-loss / critical-window locus of stability distinct from architecture, optimizer, and parametrization. In ultra-sparse MoE (top-8-of-768 at 60B/180B), lower-layer routed experts silently die: weight norms fall to ~1/10 of init while loss and load-balance dashboards look normal, and masking them barely changes quality. Diagnoses the mechanism as shallow-expert AdamW second-moment falling ~3 orders of magnitude below deeper layers — well below the conventional ϵ=10−8\epsilon=10^{-8}, which then caps the adaptive update while decoupled WD keeps shrinking parameters. Shows Muon’s ϵ=10−7\epsilon=10^{-7} (Frobenius-norm scale) is comparable to AdamW ϵ ⁣≈ ⁣10−11\epsilon\!\approx\!10^{-11} (per-coordinate), a mechanistic complement to Controlled LLM Training on Spectral Sphere‘s “Muon is half-aligned with µP” scale argument. Numerical fixes rescue norms but not downstream tasks — motivating LLAL (auxiliary LM-head loss on an early MoE layer during the first ~3% of training), which prevents collapse and persists after removal, yielding +2.8 MMLU / +4.8 MMLU-Pro at 180B.
  • [2026-08-20] Marin scaling laws simulate the entire training trajectory of a 67B MoE run within 1%: Marin’s 67B MoE run hits target loss within 1% and matches predicted per-step loss within 1% — implicit evidence that the Delphi + Quantile Balancing + AdamH stack suppresses the transient instabilities that typically break industrial MoE runs; the same ladder is now forecasting a 535B MoE trajectory in flight.
  • [2026-08-20] GEN-0: Embodied Foundation Models That Scale with Physical Interaction: GEN-0 (Generalist AI, Nov 2025) reports LLM-style ossification in robotic policy pretraining at ~O(1B) parameters — smaller models lose the ability to absorb new interaction data over the course of training, while 7B+ models continue to improve. First filed observation of the phenomenon in the embodied-model regime and at three orders of magnitude larger scale than prior LLM literature (O(10M)), positioned as evidence that stability thresholds for physical commonsense sit at a higher compute activation threshold than for abstract reasoning.
  • [2026-08-17] Training Dyna-2 at million-hour scale, repeatably: Dyna Robotics’s infrastructure companion to the 1M-hour Dyna-2 run supplies a concrete production distributed-Muon recipe for the page’s “framework-default Muon lacks FSDP/HSDP support” open question: topology-aware sharding that picks per-node NVLink hybrid sharding (with node-level duplication) above a node-count threshold and world-sharding below it, reporting ~3× speedup at scale over world-sharding. The mean-vs-median measurement (fully sharded looks competitive at median but is 2.6× worse on the mean step time due to 7.6× broadcast traffic on InfiniBand vs NVLink, and synchronous training pays wall-clock as the sum of all steps) is the specific quantification the distributed-Muon thread has been circling.
  • [2026-08-16] Cracks in the Foundation: Seemingly Minor Architectural Choices Impact Long Context Extension: Bertsch et al. (CMU / Ai2) add a long-context-extensibility locus to the architecture-side stability framing: four “minor” dense-Transformer choices — normalization variant, GQA, pretraining context length, sliding-window attention — each drawn from an Olmo/Llama/Qwen production decision, individually barely move short-context loss but combine compoundingly negatively, dropping long-context downstream performance up to 47% when 3+ combine. Short-context loss and validation are blind to these differences. The OlmPool release (26 comparable 7B models, 170k+ GPU-hours, pre/post-extension checkpoints) contains configurations that beat Llama 3 on long-context extensibility, and attention-sink and attention-distribution analyses attribute specific long-context degradations to specific choices — a mechanistic follow-on to A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training‘s “sinks are load-bearing rescaling” claim. Practical: apply context extension early in pretraining as a cheap long-context-extensibility diagnostic.
  • [2026-08-14] Breaking the Softmax Bottleneck: A High-Rank RNN Language Model: Backfill of the seminal “Softmax bottleneck” paper (Yang, Dai, Salakhutdinov & Cohen, 2017) that both Lost in Backpropagation: The LM Head is a Gradient Bottleneck and Universal One-third Time Scaling in Learning Peaked Distributions build on. Formalizes LM as matrix factorization: log-probability matrix has rank ≤ embedding dim dd, so if the true matrix is high-rank (as natural language is hypothesized to be), Softmax with d≪∣V∣d \ll |V| cannot represent it. Mixture of Softmaxes fixes the expressivity side; the 2026 papers show the same dd-vs-VV mismatch also produces optimization (rank-dd Jacobian) and scaling-law-slope (1/3 exponent) pathologies at the same layer. Historical anchor for the “LM head as structural primitive” framing on this page.
  • [2026-08-13] Dion3: Full-Stack Orthogonal Updates: Dion3 (Amsel, Zhang, Ahn, Naeimi, Feng, Chen, Dao, Langford) consolidates three separately-filed lines of Muon-overhead work into one drop-in optimizer: Gram Newton-Schulz for algorithmic FLOP reduction, CuteDSL symmetric-GEMM kernels, megabatching for communication overhead, and a new row-selection update rule that strictly improves on Dion/Dion2’s compression schemes. Claims up to 6× optimizer step-time reduction over Muon while matching or improving loss — the strongest end-to-end “distributed Muon actually affordable” datapoint on this page yet, and the convergence point of the Gram Newton-Schulz: A Fast, Hardware-Aware Newton-Schulz Algorithm for Muon and microsoft/dion — distributed Muon, Dion2, Dion, and NorMuon orthonormal optimizers threads.
  • [2026-08-12] MAGI-2 Preview: Scaling Video Generation Models Efficiently: Sand.ai MagiMuon extends the Muon-family optimizer to head × expert matrix batches from Ultra-Fine-Grained MoE: router gates batched by head, expert up/gate/down projections flatten head-and-expert into the batch dimension, Muon orthogonalization runs independently per matrix with the matrix batch distributed across ranks. Adam is retained for adapters, mHC, attention sinks, and gating parameters. Concurrent with Kimi K3’s Per-Head Muon (attention Q/K/V head split); different structures, same principle — explicit head structure in the model should be reflected in the optimizer.
  • [2026-08-11] Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization: Formal Hyperball paper (Wen, Dang, Lyu, Ma, Liang) — the arxiv writeup of what MuonH (hyperball) sets new modded-NanoGPT speedrun record at 3325 steps — Keller Jordan announcing @wen_kaiyue's result announced and Delphi: Scaling Laws That Extrapolate 300× Past the Fit deployed. Wraps any base optimizer (Adam, Muon, NorMuon, KL-SOAP) with Frobenius-sphere projection on weights + unit-Frobenius updates; on 1B Qwen3 pretraining, MuonH gives 1.4–1.6× token-equivalent speedup over MuonW with the gap growing over training. On modded-nanogpt Track 3, AdamH/MuonH/NorMuonH/KL-SOAP-H all beat their WD counterparts, with KL-SOAP-H setting the fastest step-count on the leaderboard — the strongest optimizer-agnosticism claim yet. Theory (§4) derives the mechanism from rotational equilibrium: WD’s two coupled jobs (setting equilibrium radius ρ* and setting angular step η/ρ*) get severed by pinning ρ directly, so η becomes an explicit angular learning rate. Companion Slack observation from @wen_kaiyue: Muse Glimmer’s per-matrix weight RMS is pinned at ~6e-3 = 0.5/√6656 — suggests a frontier open release (Introducing Muse Glimmer: An Open Agentic Model That Runs on Your Device) may already be running Hyperball-shaped optimization in production.
  • [2026-08-06] Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks: Kosson, Messmer & Jaggi (2023) extend SMD (Spherical Motion Dynamics: Learning Dynamics of Neural Network with Normalization, Weight Decay, and SGD) from SGDM to AdamW / Adam+L2 / Lion and give the prescriptive half of the story: build “Rotational Variants” that project to the equilibrium ‖w‖ and enforce the equilibrium Δθ explicitly, dispensing with weight decay tuning. Central mechanistic claim: AdamW beats Adam+L2 (and Weight Standardization beats vanilla LayerNorm) because AdamW’s equilibrium rotation depends only on hyperparameters while Adam+L2’s depends on gradient magnitude — so identical HPs produce ~30× per-layer rotation variation (≈1000× effective-LR variation) in Adam+L2 but a tight band in AdamW. Direct theoretical ancestor of Modular Manifolds, Controlled LLM Training on Spectral Sphere, Delphi: Scaling Laws That Extrapolate 300× Past the Fit‘s AdamH, and Introducing Inkling — Thinking Machines' open-weights 975B/41B MoE multimodal reasoning model‘s WD∝LR² coupling. Caveat matching the filing pointer: RV matches baseline AdamW zero-shot on 5/6 configs — impressive as theory validation, not a benchmark win.
  • [2026-08-06] Spherical Motion Dynamics: Learning Dynamics of Neural Network with Normalization, Weight Decay, and SGD: Backfill of the SMD foundation the rest of the rotational-equilibrium / weight-geometry cluster on this page builds on. Wan, Zhu, Zhang & Sun (2020) prove that for scale-invariant weights under SGDM+WD the weight norm converges to an equilibrium at linear rate long before the loss converges, and derive the closed-form angular update Δθ ≈ √(2ηλ/(1-β)) that depends only on hyperparameters. This is the mechanistic root of How to Set the Learning Rate for Large-Scale Pre-training?‘s “per-module AdamW updates ≈0.2 across all layers” observation and of the whole SSO / AdamH / modular-manifolds / Inkling-WD∝LR² line — but Kyle’s team-side flag (per the filing pointer) is worth carrying forward: the framework studies angular updates in isolation and its practical gains in the original paper are equilibrium-recovery hacks (rescale ‖w‖ ← ‖w‖·√α at LR drops) rather than headline accuracy improvements.
  • [2026-08-05] CMuon: Accelerating and Stabilizing Diffusion Transformer Training via Chunked Momentum Orthogonalization: CMuon (Chen, Sun, Yuan) locates a DiT-specific Muon pathology that the LLM-side literature does not surface: standard DiTs fuse AdaLN (scale/shift/gate) and QKV projections into single tensors for GEMM efficiency, and Muon’s per-tensor Newton–Schulz orthogonalization treats the fused tensor as one operator — inducing implicit subspace coupling that distorts updates and produces vanilla Muon’s late-stage FID plateau on DiTs. Fix is a one-line decoupling: chunk the fused matrices along their functional boundaries before msign, orthogonalize each chunk independently, re-concatenate. Reports 675M DiT / ImageNet 256² → FID 1.18 in 200 epochs, >2× over AdamW, past vanilla Muon’s plateau. Third distinct Muon-pathology paper on the wiki alongside SSO (weight-spectrum half-alignment) and Aurora (row-leverage on rectangular matrices), and the first one whose failure mode is architecture-fusion-specific rather than optimizer-internal — reads as the principled version of Krea 2’s ad-hoc “exclude first/last linears + Nesterov” Muon workaround on MMDiT.
  • [2026-08-01] Multi-Head Attention Residuals: Multi-Head Attention Residuals (MHAR) is the multi-head lift of Moonshot’s Attention Residuals: reshaping the single per-layer routing query into H per-subspace heads so different feature subspaces can softmax-attend to different prior layers, with a U-shaped optimum at H=4 or H=8 across 100M–1B scales, zero extra parameters, and fused Triton kernels bringing training throughput from 0.2–0.5× to 0.55–0.88× of a plain-Transformer baseline. Delta-attention-residual mid-training at 8B is identity-preserving and yields +3.2 GSM8K / +3.1 GPQA — the AttnRes lineage now has a low-friction mid-training upgrade path.
  • [2026-07-30] A Defense of the Quadratic Model: Meterez, Nair, Morwani, Pehlevan, Kakade, Damian defend the local-quadratic model of LLM optimization: at 150M params / 3B tokens, Taylor-expanding the loss (quadratic) and model (linearized) at intermediate checkpoints reproduces the real training trajectory over windows lasting up to ~10% of training, with better agreement late in training and under cosine than under constant-LR. Deep Lanczos quadrature (m=1200) resolves the Hessian spectrum across six orders of magnitude, revealing a vocab-size head of eigenvalues (neural collapse) + a universal power-law tail that is invariant to batch size and to Adam preconditioning. Local linear-stability probes show LLM training operates at a stochastic EoS at small batch and a deterministic EoS at large batch, matching linear-stability theory — giving a data-side complement to the architecture/optimizer/parametrization loci already tracked here: whatever knobs you turn, the underlying loss is well-approximated by a structured quadratic and training sits on its stability boundary.
  • [2026-07-30] The Obsessed Encoder — Latent-Space Misallocation in JEPA-Style Self-Supervised Models: Enigma reports that the load-bearing anti-collapse regularizers of modern JEPA-style SSL (SIGReg’s Cramér-Wold + Epps-Pulley Gaussianity test in LeJEPA / LeWorldModel; DINOv3’s Sinkhorn-Knopp centering + KoLeo nearest-neighbor entropy) are silently defeated by low-entropy predictable features — an expressive encoder can hash a 3-DoF manifold into 192-D so every random 1-D projection matches N(0,1)\mathcal{N}(0,1), and KoLeo continues to spread neighbors apart in the hashed space. A stability-locus datapoint distinct from the architecture / optimizer / parametrization framings tracked on this page: Gaussianity-of-projections and nearest-neighbor entropy are not load-bearing collapse detectors when the encoder is expressive enough to hash a low-D signal, which suggests SSL anti-collapse mechanisms may need to bound intrinsic dimensionality rather than marginal shape.
  • [2026-07-28] Online KL Shampoo (OKLS) — Tilde Research's KL-optimal full-matrix AdaGrad approximation optimizer with 1.45× parameter efficiency over Muon: Tilde Research’s third 2026 optimizer, Online KL Shampoo (OKLS), attacks a different pathology than Aurora (row leverage) and Compositional Muon (QK/OV compositions): the history of gradient correlations that Muon deletes each step by re-orthogonalizing. Approximates full-matrix AdaGrad via KL-optimal Kronecker factors (novel vs Shampoo/Dion’s Frobenius-optimal factors) with zero-staleness inverse-square-root preconditioning; claims 1.45× parameter efficiency over Muon at 98% throughput across 200M–1B parameters. The three-way Tilde cluster (Aurora / CM / OKLS) now asks whether disjoint-pathology fixes compose or saturate the same optimizer headroom.
  • [2026-07-27] Delphi: Scaling Laws That Extrapolate 300× Past the Fit: Delphi’s AdamH optimizer intervention (Kaiyue Wen, Marin) is a stability lever framed as removing weight decay from the hyperparameter search: every Adam step is followed by a Frobenius-norm-sphere retraction that first takes a step of Frobenius norm matching η∥W∥F\eta \|W\|_F regardless of the raw update magnitude, then projects back to the sphere of radius ∥W∥F\|W\|_F so the weight norm is preserved exactly. There is nothing left for weight decay to regularize away, and in the Delphi width-scaling replication the best learning rate stays more bounded across widths under AdamH than under Adam. Sits alongside SSO (Controlled LLM Training on Spectral Sphere, per-module spectral sphere) and the Inkling weight-decay-∝-LR² coupling rule (Introducing Inkling — Thinking Machines' open-weights 975B/41B MoE multimodal reasoning model) as three flavors of the same idea: constrain the geometry of weights (Frobenius sphere / spectral sphere / LR-coupled decay) so the hyperparameter search is smaller and stability is enforced by construction rather than tuning.
  • [2026-07-24] SANA-Video 2.0: Hybrid Linear Attention with Attention Residuals for Efficient Video Generation: First external adoption of Moonshot’s Block Attention Residuals (Attention Residuals (AttnRes / Block AttnRes) — Technical Report) outside Kimi Linear — SANA-Video 2.0 uses AttnRes to route completed block summaries into later linear layers of a 5B/14B video DiT and reports the same PreNorm-dilution signature Moonshot documented on LLMs: ~12% lift in deep-layer effective rank. Confirms AttnRes generalizes from LLM stacks to diffusion transformers.
  • [2026-07-16] Kimi K3 — Open Frontier Intelligence (2.8T MoE with KDA + AttnRes): Kimi K3 pairs Per-Head Muon (Muon with per-attention-head decoupling) with QAT-from-SFT in MXFP4 weights + MXFP8 activations and reports stable training at 2.8T parameters with 16/896 expert sparsity.
  • [2026-07-15] Introducing Inkling — Thinking Machines' open-weights 975B/41B MoE multimodal reasoning model: Inkling’s public training recipe: hybrid Muon (matrix weights) + Adam (everything else) with weight decay coupled to LR² — the LR²-coupling is claimed to keep weight-matrix norms stable across training horizons, letting the same recipe transfer across LR schedules. First frontier-scale from-scratch pretraining run to name modular-manifolds-style spectral control + LR²-coupled decay as its stability primitive; 45T-token, 975B/41B MoE run on GB300 NVL72.
  • [2026-07-13] Modular Duality in Deep Learning: Bernstein & Newhouse’s foundational duality-map paper — the theoretical origin of the rectangular Newton-Schulz iteration that Muon uses to orthogonalize its momentum. Derives per-layer duality maps from operator norms (RMS→RMS for Linear, ∞→RMS for Embed, max-over-kernel for Conv2D) and composes them via the modular norm’s max-weighted composition, giving straightforward all-layer updates. Shows µP and Shampoo both fall out as partial approximations to the single √(d_out/d_in)·UVᵀ duality map for Linear — reframing “should we tune per-module LRs?” as “which pieces of this duality map does the optimizer actually implement?” Grounds the SSO claim that Muon is “half-aligned with µP” in a precise object.
  • [2026-07-10] The 4-bitter Lesson: Balancing Stability and Performance in NVFP4 RL: Dequantized backward as a new stability-under-quantization lever: replacing y = x·w_bf16 in the BF16 backward with y = x·DQ(Q(w_bf16)) matches the chain rule to the actual quantized forward, eliminating the grad-norm spikes that appear under the “quantized forward, higher-precision backward” baseline. Under Adam’s momentum + adaptive second moment the extra NVFP4 gradient variance is absorbed, yielding far fewer large spikes than BF16 backward. Combined with 4/6 and selective precision, NVFP4 RL matches BF16 grad-norm across five runs where any two-out-of-three combination still spikes.
  • [2026-07-09] Scaling with Collapse: Efficient and Predictable Training of LLM Families: Bergsma et al. deploy scaling-collapse residuals as a production stability signal on a Cerebras 300M–3.9B (Celerity) family: at 1.8B, collapse residuals surfaced a numerical instability at ~60% of training, whereas the raw loss trend only revealed it at ~90%. Ablations against the collapse reference isolated the bug to a microbatch-size-triggered loss-kernel numerical issue and enabled a targeted restart. Concrete demonstration that the Qiu et al. supercollapse diagnostic operationalizes at LLM scale as an earlier-and-sharper warning than trend-watching. Also documents CompleteP outperforming µP for joint width–depth scaling on this family, adding a data point to the µP-vs-CompleteP debate elsewhere on the page.
  • [2026-07-09] Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks: Qiu et al. establish scaling collapse and supercollapse as a live diagnostic: mis-parameterization (dropping µP for a constant LR across widths) breaks collapse even at small scales where final losses are indistinguishable. Adds a new class of stability signal to the page — collapse residuals give an earlier, more sensitive warning of scaling-parametrization pathologies than final-loss comparison or trend-watching alone. Complements Training stability at scale‘s architecture-vs-optimizer-vs-parametrization framing with a fourth “does dynamics stay consistent across scales?” axis.
  • [2026-07-08] LingBot-Video: Scaling Mixture-of-Experts Video Pretraining for Embodied Intelligence: LingBot-Video’s stability recipe for scaling a MoE video DiT to 120B/11B-active: QK-Norm (per-head RMSNorm on Q/K before attention), adaLN-single modulation (one shared timestep MLP + per-layer trainable modulation tables, zero-init on shared projection), auxiliary-loss-free bias for MoE load balancing plus sequence-wise Lseq to prevent per-video routing collapse, and √(d/d_a)-style variance-preserving expert init.
  • [2026-07-01] To Use or not to Use Muon: How Simplicity Bias in Optimizers Matters: Dragutinović et al. attack Muon from a new angle — not stability, not µP-alignment, but inductive bias. Under an exact-SVD, no-momentum abstraction of Muon (Spectral GD) on 2-layer deep linear networks, they prove GD’s classical saddle-to-saddle dynamics sequentially increase solution rank (largest singular value first) while Spectral GD learns every singular component in parallel at equal rate — losing GD’s implicit rank-based regularization. Two constructed head-to-heads have SGD beat Muon: (a) a multi-source “routing” task where SGD discovers a shared 4-D representation and generalizes to unseen source pairs while stochastic Spectral GD memorizes each pair despite matching train loss, and (b) MNIST + per-class spurious pixel where SGD’s dominant-first learning gives higher clean-digit accuracy up to spurious intensity ≈2. Also observes that after the theoretical low-rank modes are learned, Spectral GD’s discretization noise makes the update full-rank and it starts fitting that noise in a highly oscillatory trajectory that momentum does not damp — a mechanism-level hint that Muon may be less robust at high learning rates in richer settings than the current stability literature captures.
  • [2026-06-30] What do near-optimal learning rate schedules look like?: Naganuma et al. find that on the small WikiText-103 Transformer workload, the most aggressive near-optimal LR shapes produce some training-instability blowups (large-perplexity runs) that inflate confidence intervals, echoing the broader story that the optimal shape sits near the stability boundary — and on linear regression the analytical optimum is exactly at the edge-of-stability. AdamW weight decay (a known stability knob) is the strongest covariate of optimal shape, supporting the architecture/optimizer-side stability framing on this page.
  • [2026-06-24] Krea 2 Technical Report: Krea 2’s architecture-stability moves on a production T2I MMDiT: GQA + sigmoid-gated attention (small wall-clock cost, no quality gain but visibly smoother loss and gradient-norm curves), zero-centered RMSNorm with weight decay on its learnable params, QKNorm, per-block tunable bias replacing AdaLN MLPs. Reports Muon (Dion implementation + Moonlight RMS-matched HP transfer) is stable enough to beat AdamW at low and high resolution only after excluding the first and last linear layers and adding Nesterov momentum — otherwise it triggers loss/gradient-norm spikes.
  • [2026-05-26] Laguna M.1/XS.2 Technical Report: Laguna documents the M.1→XS.2 stability redo: keep the first Transformer layer dense, FP32 RMSNorm/RoPE, configurable gradient-reduction dtypes per collective (FP32 default for sensitive paths including the LM-head input-gradient all-reduce), and a WSD schedule with the peak chosen from an N,D scaling law rather than a one-shot cosine tune.
  • [2026-05-24] Addition of Muon optimizer to torch.optim — PyTorch team welcomes PR (Issue #148819): PyTorch maintainers’ July-2025 comment welcoming a PR to add Muon to torch.optim, explicitly citing Moonshot’s MuonClip / Kimi K2 deployment as the social proof. Marks the point at which Muon transitioned from “research-lab optimizer” to a framework default. The infrastructure caveat — distributed Muon (FSDP/HSDP) is still not supported in core, and Kimi K2 disables FP8 GEMM for the Newton–Schulz step — is the open piece the SSO line (Controlled LLM Training on Spectral Sphere) and Aurora (Aurora: A Leverage-Aware Optimizer for Rectangular Matrices) are addressing.
  • [2026-05-23] Quantile Balancing: A Hyperparameter-Free MoE Load Balancing Method: hyperparameter-free MoE load balancer (QB) from Jianlin Su, formulated as a per-step LP over bias offsets and solved via alternating quantile iteration. Sits in the same lineage as MuonClip / qk-clip — “eliminate the tuning knob entirely” — but applied to the router rather than to attention logits. Validated by the Marin team at 32B-A5B / 1e22 FLOPs (326B tokens) with zero loss spikes; 1e23-FLOP run underway. Suggests MoE routing is another locus where the “constrained optimization replaces hand-tuned heuristic” pattern works.
  • [2026-05-23] Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning: introduces µA (Maximal-Update Adaptation), the LoRA analogue of µP, characterizing optimal-LR scaling in the joint n,r→∞n, r \to \infty limit. Identifies Init[B] + α=1\alpha=1 as the unique LoRA configuration whose optimal LR coincides with full-finetuning’s, enabling direct LR transfer. Provides a finetuning-side counterpoint to 2601.05049: µP-style asymptotic analysis is still load-bearing here, even though from-scratch LLM pretraining no longer needs it under QK-Norm. Verified across SFT (5 models), RLVR, and Stable Diffusion.
  • [2026-05-23] ViT-5: Vision Transformers for The Mid-2020s: vision-side replication of the architectural-stability story — QK-Norm in ViTs suppresses sharp loss spikes during optimization, mirroring the LLM-side finding. Adds a cross-modality negative result: SwiGLU + LayerScale together over-gate ViTs (both are channel-wise filters; combined sparsity hurts), so the LLM→ViT port has to drop SwiGLU even though every other component transfers. Hits ImageNet-L/384² SoTA (86.0%) and SiT-XL FID 1.84.
  • [2026-05-23] Controlled LLM Training on Spectral Sphere: introduces SSO, an optimizer that constrains both weights and updates to a per-module spectral sphere via steepest descent under the spectral norm + Newton–Schulz msign + retraction. Argues Muon is “half-aligned” with µP (controls updates, not weights) and that closing this loop gives strict Θ(1)\Theta(1) activation control end-to-end. Outperforms AdamW and Muon on Dense 1.7B, MoE 8B-A1B, and 200-layer DeepNet; ships as a Megatron-LM fork with Atomic Module Sharding and cached singular vectors. Directly conflicts with 2601.05049 on whether µP’s motivation still stands.
  • [2026-05-23] Universal One-third Time Scaling in Learning Peaked Distributions: derives a universal 1/3 time-scaling exponent for cross-entropy loss when the target distribution is peaked, attributing it to softmax saturation and the resulting power-law-vanishing gradient. Reframes most pretraining interventions (Muon, data mixes, normalization) as intercept-only shifts of the scaling law — its slope is structural.
  • [2026-05-23] A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training: unifies attention sinks and residual sinks as outlier-driven rescaling; replaces the implicit mechanism with explicit Gated Attention / GatedNorm, yielding pretraining gains and W4A4 quantization robustness. Locates a previously-unexamined coupling between training stability and low-precision friendliness.
  • [2026-05-22] How to Set the Learning Rate for Large-Scale Pre-training?: shows µP’s motivating attention-logit instability is suppressed by QK-Norm under SP, so module-specific LRs no longer pay off and the Fitting paradigm (scaling-law extrapolation of LR*(N,D)) beats µTransfer at 4B and 12B MoE scale.
  • [2026-05-22] mHC: Manifold-Constrained Hyper-Connections: makes residual-stream geometry the load-bearing stability primitive by constraining the residual-mixing matrix to the Birkhoff polytope, suppressing 27B-scale loss spikes and bounding composite signal gain at any depth.

Open questions

Section titled “Open questions”
  • All three architectural-stability papers locate stability in different places in the network — Q/K projections (QK-Norm), attention output + RMSNorm (GatedNorm / Gated Attention), residual mixing (mHC). Are these complementary (stack them) or do they substitute for one another at sufficient strength? No filed paper has run the stacked ablation.
  • Architecture-side vs optimizer-side stability fixes are now an explicit fork. SSO (Controlled LLM Training on Spectral Sphere) suppresses outliers and bounds activations as a side-effect of spectral retraction; Gated Attention (A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training) preserves outliers but adds explicit learned rescaling; mHC bounds composite signal gain at the residual-mixing level. Do these stack (SSO + Gated Attention + mHC) or do they each saturate the same underlying degree of freedom (bounded effective spectra of the forward/backward map)? No filed paper has done the cross-comparison.
  • Pretraining vs finetuning may be a real split. How to Set the Learning Rate for Large-Scale Pre-training? declares µP’s stability motivation stale for from-scratch pretraining under QK-Norm; Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning shows µP-style analysis remains predictive for LoRA finetuning. Whether this means “stability is solved by architecture, but rank/init/multiplier coupling needs separate parametrization analysis” or “µP’s relevance is regime-dependent in a way no filed paper has fully mapped” is open.
  • The Gated Attention story explicitly couples stability with quantization-robustness — outliers are both why training works and why PTQ fails. Does the same coupling apply to mHC’s residual-stream constraint (does bounding composite gain to ~1.6× also make activations PTQ-friendly)? SSO claims the same coupling (bounded activations → low-precision friendly). The mHC paper does not measure this; the SSO paper asserts it without quantifying PTQ deltas.
  • µP was designed for LLM decoder-only stacks. Do these architecture-first stability fixes transfer to DiT / MMDiT pretraining, where AdaLN modulations are an additional outlier source and the residual streams carry conditioning rather than just token state? Untested across all five filed papers — though ViT-5: Vision Transformers for The Mid-2020s supplies a partial answer for plain SiT backbones (QK-Norm transfers cleanly to vision; SwiGLU does not) and Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning does run Stable Diffusion finetuning (Init[B] + α=1\alpha=1 optimum holds; FFT is sharply LR-sensitive on UNets). The SSO Megatron fork is the lowest-friction way to test the optimizer-side hypothesis on a DiT codebase.
  • Cross-modality component transfer is not uniform: ViT-5 finds QK-Norm, RMSNorm, LayerScale, RoPE, and bias-free-QKV all transfer from LLMs to ViTs, but SwiGLU does not (over-gates with LayerScale). Is there a similar transfer asymmetry going the other direction — components that work in ViTs but not LLMs? The fact that LayerScale ≈ post-RMSNorm in ViTs (ViT-5: Vision Transformers for The Mid-2020s §3.1, Table 1) but LayerScale is rare in LLMs raises this question directly.
  • The Fitting paradigm in How to Set the Learning Rate for Large-Scale Pre-training? fits LR*(N, D) under WSD. If the architectural stability story is right, the LR* surface should become flatter under stronger normalization (less penalty for slight LR overshoot). Could be a clean prediction to falsify. SSO’s claim that µP transfer still holds under spectral-sphere constraints is a related, testable prediction.
  • All papers run under LLM-style pretraining (decoder-only, large-N MoE). Do the same instabilities (sinks, identity-violation, attention-logit blowup, 1/3-exponent slope) appear at the smaller-N high-batch regime that diffusion-from-scratch pretraining actually lives in? The 1/3 paper makes a specific prediction here: MSE / flow-matching objectives should not exhibit the softmax-CE bottleneck and so the DiT slope should be in a different universality class.
  • If the 1/3 exponent is structural to softmax-CE on peaked targets, does it survive when the target distribution is broadened (label smoothing, mixture-of-token-distributions distillation, temperature-scaled teachers)? A clean falsifier for whether the slope can be moved.
  • Is “Muon is half-aligned with µP” the right framing, or does Muon’s implicit weight-spectrum control via update orthogonalization already deliver most of SSO’s gains? SSO’s MuonSphere variant ablates exactly this — claim is the gap matters, but the precise delta between Muon and MuonSphere is the load-bearing experiment for this whole line.
  • Does QB’s LP framing generalize to other discrete-routing decisions beyond MoE expert assignment — e.g. mixture-of-depths gating, conditional-compute heads, layer-skip routing? QB’s claim is that the routing target (uniform load) is what makes the LP solvable in closed-quantile form; whether other targets (e.g. accuracy-weighted load, capacity-aware load) admit similar one-shot solutions is open.
  • How does QB interact with the SSO finding that spectral-sphere training “improves MoE router load-balancing” as a side-effect? If the optimizer already bounds router activations and reduces outlier-driven imbalance, do you still benefit from QB on top, or do they substitute? Both touch MoE balancing but from different loci (optimizer vs. routing-time correction).
  • What does framework-default Muon (no distributed support) imply for adoption? Per Addition of Muon optimizer to torch.optim — PyTorch team welcomes PR (Issue #148819), torch.optim.Muon shipped without FSDP/HSDP support, and Kimi K2’s deployment explicitly disables FP8 GEMM for the Newton–Schulz step. The follow-on optimizer papers (SSO, Aurora, Newton-Muon) increasingly target the implementation gap — distribution, FP8, leverage-awareness — rather than the algorithm itself. Whether one of these becomes the canonical “distributed Muon” recipe in torchtitan is the live infra question.

Papers

Section titled “Papers”

2026-05

Section titled “2026-05”

2026-04

Section titled “2026-04”

2026-03

Section titled “2026-03”

2026-02

Section titled “2026-02”

2026-01

Section titled “2026-01”

2025-12

Section titled “2025-12”

2025-11

Section titled “2025-11”

2025-10

Section titled “2025-10”

2025-09

Section titled “2025-09”

2025-08

Section titled “2025-08”

2025-07

Section titled “2025-07”

2025-06

Section titled “2025-06”

2025-05

Section titled “2025-05”

2025-04

Section titled “2025-04”

2025-03

Section titled “2025-03”

2025-02

Section titled “2025-02”

2025-01

Section titled “2025-01”

2024-01

Section titled “2024-01”

2020-12

Section titled “2020-12”

Unknown date

Section titled “Unknown date”