Mitigate Silent Expert Death In Ultra-Sparse MoE

ListenSilent Expert Death: When Half Your MoE Is Already Gone10:43

Show notes

A deep dive into a Notion post from Hongye Jin and colleagues documenting **silent expert death** in ultra-sparse mixture-of-experts pretraining — where lower-layer experts quietly go inert while every dashboard metric looks perfectly healthy. We unpack the mechanism (a collapsed learning signal colliding with AdamW's epsilon and weight decay), why Muon and small-epsilon AdamW rescue the norms but not always the quality, and the authors' fix: **LM Loss as Auxiliary Loss (LLAL)**. Plus the audit that names frontier open models showing the same signature. Sources - Mitigate Silent Expert Death In Ultra-Sparse MoE — Hongye Jin, Linwei Li, Xiaotian Han, Xin Liu, Haoyang Wen, Tuo Zhao, Qingyu Yin, Binxuan Huang

Subscribe to the channel: https://podcast.lumalabs.link/27a25f51ea81d41a8253f46c6133a35e/feed.xml

TL;DR

A long-form Notion post from Hongye Jin et al. documenting a silent failure mode in ultra-sparse MoE pretraining: at top-8-of-768 routing on 60B and 180B models, lower-layer routed experts progressively lose weight norm (down to ~1/10 of init) and become functionally inert, while training loss, validation loss, and load-balance metrics all look healthy. Masking every routed expert in a collapsed layer barely changes model quality — the layer has degenerated into a dense (shared-expert-only) or null layer. The authors trace the mechanism to a collapsed learning signal (per-coordinate second-moment falling far below AdamW’s $10^{-8}$ epsilon while decoupled weight decay keeps shrinking parameters), show that Muon and small-epsilon AdamW ( $\epsilon = 10^{-12}$ ) both rescue the norm collapse for a scale-comparable reason (Muon’s $10^{-7}$ epsilon acts at the Frobenius scale of an entire momentum matrix rather than per-coordinate), and introduce LM Loss as Auxiliary Loss (LLAL): for a few thousand steps early in training, attach an auxiliary LM head to an early MoE layer so upper layers cannot cover for it. At 180B / 135k steps, using LLAL for only 4k steps prevents collapse, improves validation loss and load balance, and raises MMLU and MMLU-Pro by 2.8 and 4.8 points respectively. An audit of open-source MoEs finds similar early-layer collapse signatures in MiMo-V2.5-Pro, Qwen3.5-397B-A17B, and Qwen3.8-Max, and in Kimi-K3 the learned load-balancing bias overrides even the router’s top-1 choice on >70% of tokens in some layers.

Key claims

Ultra-sparse MoE training exhibits silent expert death: at top-8-of-768 routing, routed-expert weight norms in lower MoE layers fall below ~1/10 of their initialization, yet training loss, validation loss, and standard load-balance metrics all look healthy [§1 “Silent Expert Death”; Fig. 1, Fig. 2 in Section 1 reproduction].
Masking every routed expert in a collapsed layer produces almost no degradation in downstream model quality — the layer has effectively degenerated into a dense layer (shared expert only) or a null layer [§2.1 “Masking experiments”].
The eventual layer-wise expert utilization pattern is determined within the first few thousand training steps; after this critical window, downstream modules stop depending on the lower pathway and its learning signal collapses [§2.1; §3.4].
Shallow-expert AdamW second-moment estimates fall by ~3 orders of magnitude below deeper-layer values, taking the second-moment scale far below the conventional $\epsilon = 10^{-8}$ ; in this regime $\epsilon$ no longer merely prevents division-by-zero but caps the adaptive update, while decoupled weight decay continues shrinking the parameter [§3.2].
Muon and AdamW-with- $\epsilon=10^{-12}$ produce similar expert-norm dynamics for a scale reason: Muon applies its $\epsilon$ to the Frobenius norm of a whole momentum matrix before orthogonalization, so its commonly used $10^{-7}$ has an intervention scale closer to AdamW’s $10^{-11}$ than to $10^{-8}$ for the expert-matrix shapes used [§3.3].
Numerical fixes (small AdamW $\epsilon$ or Muon) rescue weight norms but do not always improve downstream metrics — healthy norms are not sufficient for useful experts [§3.2, §3.3].
LLAL (LM Loss as Auxiliary Loss) — temporarily attach the LM head to an early MoE layer and add its LM loss as an auxiliary term during the critical early window, then remove — prevents norm collapse, improves validation loss and load balance, and persists after removal because the main loss now finds and preserves the lower-pathway dependency [§3.4].
At 180B parameters over 135,000 training steps, LLAL applied for only 4,000 steps raises MMLU by +2.8 and MMLU-Pro by +4.8 points, alongside better validation loss and markedly better load balance [§3.4, “180B experiment”].
Router learning rate, load-balancing scope, Megatron gradient-buffer semantics, and residual design each independently reshape expert learning dynamics beyond LLAL [§Beyond LLAL; Section 3.1–3.4].
Open-source MoE audit: MiMo-V2.5-Pro, Qwen3.5-397B-A17B, and Qwen3.8-Max exhibit collapsed-layer signatures similar to the paper’s own runs; Qwen3.8-Max additionally shows two large blocks of layers with high mutual similarity, suggesting possible layer copying [Appendix F].
In Kimi-K3, the learned auxiliary-loss-free load-balancing bias overrides the router’s own top-1 expert choice on >70% of tokens in some layers, occasionally promoting experts ranked below 400 by the router’s raw score [Appendix F].

Method

The paper is written as an experimental narrative rather than a single-method proposal. The core diagnostic setup is a 60B-parameter model (32 layers, MLA, 16 heads; first layer dense, remaining 31 as MoE with 768 routed + 1 shared expert, top-8 sigmoid routing, DeepSeek-style auxiliary-loss-free load balancing with fixed bias update rate 0.001, SwiGLU activation) deliberately made “substantially sparser, narrower and deeper” than mainstream designs so that the failure mode is easy to trigger. Baseline (Exp1): batch size 384, ~280B tokens, cosine LR schedule with 2000 warmup steps over 90,000 steps, AdamW with weight decay 0.1. A 5B / 32-layer / 512-expert / top-8 variant is used for small ablations, and one 180B experiment closes the story.

Three intervention families are then run against this baseline: (1) optimizer/numerics, lowering AdamW $\epsilon$ (down to $10^{-12}$ ) or switching to Muon, which prevents visible norm collapse but does not always improve downstream tasks; (2) LLAL, which attaches the LM head to the output of an early MoE layer during the first ~4% of training (4k of 135k steps at 180B), adds that head’s cross-entropy as an auxiliary loss, then removes both the auxiliary head and the loss for the remainder of training; (3) auxiliary levers — router-only learning rate, load-balancing scope, Megatron’s gradient-buffer semantics, and residual-connection design — each individually studied for their effect on expert-norm dynamics.

Results

Baseline Exp1 reproduces the failure mode: first-MoE-layer expert weight norm ~1/10 of higher healthy layers by end of training, healthy training/validation curves throughout [Section 1, Fig. 2].
Small- $\epsilon$ AdamW and Muon rescue weight norms, but downstream metrics do not consistently follow — reported as evidence that norm health is not equivalent to functional utility [§3.2, §3.3].
180B / 135k-step LLAL run (4k steps of LLAL, then removed): no silent expert collapse [Fig. 4]; improved validation loss [Fig. 3]; better downstream metrics — MMLU +2.8, MMLU-Pro +4.8 [Fig. 4]; markedly better load balance [Fig. 4].
LLAL cost is bounded: auxiliary head and loss are removed after the critical window, adding no inference-time overhead and only a small compute overhead during that window.
Open-model audit (Appendix F): MiMo-V2.5-Pro, Qwen3.5-397B-A17B, Qwen3.8-Max all show collapsed early-layer signatures; Qwen3.8-Max has two similarity-blocked layer groups suggestive of layer copying; Kimi-K3’s aux-free bias overrides router top-1 on >70% of tokens in some layers.

Why it’s interesting

Slots directly into the Training stability at scale framing where architecture, optimizer, and parametrization each get named as the stability locus — LLAL adds a fourth locus, auxiliary training-time signal shaping during a critical window, that none of the current entries occupy. The paper also delivers the sharpest concrete quantification to date of a claim already implicit in Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models: that standard MoE monitoring (loss curves + load-balance metrics) is not a sufficient diagnostic. On the MoE Routing Design axis it complements the “eliminate the tuning knob” line (Quantile Balancing: A Hyperparameter-Free MoE Load Balancing Method, MuonClip / QK-Clip) with a diagnostic-first framing: before deciding whether the bias-update or aux-loss coefficient needs tuning, first check whether the experts you’re balancing are actually alive. The Muon-vs-AdamW- $\epsilon$ scale analysis in §3.3 is a valuable mechanistic complement to the Controlled LLM Training on Spectral Sphere “Muon is half-aligned with µP” argument — same optimizer pair, different failure mode, same conclusion that the numerical constants inside Muon and AdamW are not directly comparable. The Appendix F audit of open MoEs also makes this the first filed artifact to name specific frontier models (MiMo-V2.5-Pro, Qwen3.5-397B-A17B, Qwen3.8-Max, Kimi-K3) as exhibiting the collapse signature.