The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertise

TL;DR

Hayou et al. give a mechanistic account of what MoE “expert specialization” actually is: because the router is a linear projection from token hidden states to expert logits, hidden-state similarity is both necessary and sufficient to explain expert-usage similarity. Specialization is therefore a property of the hidden-state geometry, not of the routing architecture per se. The paper proves a tight data-geometry-aware upper bound on router-logit distance, formalizes how load-balancing loss suppresses shared hidden-state directions to maintain routing diversity (offering a theoretical explanation for the small-batch specialization collapse observed by Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models), and then demonstrates that despite this clean mechanistic story, actual specialization patterns in pre-trained MoEs resist human interpretation across three concrete stress tests. The conclusion: understanding expert specialization is at least as hard as understanding LLM hidden-state geometry, a long-standing open problem.

Key claims

The MoE router is a linear map from hidden states to expert logits, so token pairs with similar hidden states must activate similar experts; expert specialization is an emergent property of the representation space, not of the routing mechanism itself [§1, Prop. 1].
A tight data-geometry-aware upper bound on router-logit distance is proven (Proposition 1): projecting hidden states onto the top-r right singular vectors of the data matrix, the logit distance is bounded by the projected hidden-state distance times the router’s operator norm on that subspace, plus a residual that is negligible when data variance concentrates in the top directions [§4.1, Eq. (4), App. B].
The upper bound produces a triangle shape empirically: small hidden-state distance forces similar logits (triangle apex), but large hidden-state distance permits either similar or dissimilar logits (triangle base) — verified on OpenWebText across five pre-trained MoEs [Fig. 1].
Load-balancing loss forces the router to suppress the shared direction in correlated hidden states (Proposition 2, informal): under a simplified shared-direction + independent-residuals model of hidden states, the near-minimizer of a simplified LBL satisfies w·μ = 0 — the router ignores the shared direction because softmax + top-K are shift-invariant [§4.1, App. B.1].
This explanation predicts why small-batch LBL produces specialization collapse (as observed by Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models): small batches lack shared-direction diversity, so the shared direction the router is forced to suppress changes batch-to-batch, preventing the router from finding informative directions [§4.1].
Router alignment with the data subspace produces an OOD-detection side-effect: OOD inputs (simulated by token reversal/shuffling) show lower router confidence (maximum softmax probability) than in-distribution inputs [Fig. 3].
Expert usage is context-dependent, contradicting OpenMoE’s earlier “Context-Independent Specialization” claim: the same token in different contexts (apple in daily vs coding context; a 6-token sentence in daily vs medical context) activates overlapping experts in the first layer but progressively divergent experts in later layers on gpt-oss-20b [§4.2, Fig. 4].
Different reasoning models solving the same HMMT February 2025 problem show only ~60% expert overlap on average — comparable to the same model solving different questions, and far below the same-model same-question baseline [§5.1, Fig. 6].
Prefill-phase expert usage is not a reliable proxy for generation-phase usage: semantically different prompts activate nearly identical experts during prefilling but diverge during generation, and which regime applies is not predictable from the prompt alone [§5.2, Fig. 7].
Router collapse in later layers during prefilling: in gpt-oss-20b’s last two layers, a Humanity’s Last Exam question and a Wimbledon history passage activate an identical set of experts; pruning to top-12 experts causes negligible NLL increase for prompt-only input but is highly consequential once generation begins [§5.3, Figs. 8, 9].
Router collapse is hypothesized to be a property of structured-reasoning models: not observed in the non-reasoning ERNIE-4.5-21B-A3B-PT, and reduced (but not eliminated) when the chat template is removed [§5.3, Fig. 10, Fig. 28].
Auxiliary-loss-free models (Moonlight 16B, Trinity Mini) do not show the router-suppression-of-shared-direction pattern that auxiliary-loss models exhibit — consistent with Prop. 2’s mechanism [§4.1, Figs. 15(c), 15(d)].

Method

The paper is a mechanistic-analysis paper with two theoretical propositions and three empirical stress-tests, not a training paper. Proposition 1 starts from the observation that the router output is r = W · h where W ∈ R^(E×d) and h ∈ R^d is a token hidden state, applies the triangle inequality after projecting onto the top-r right singular vectors of the data matrix X (whose rows are token hidden states), and yields the bound ||W(h₁ − h₂)|| ≤ ||W P_r|| · ||h₁ − h₂||_{P_r} + residual. Proposition 2 models correlated hidden states as h_i = μ + ε_i with ε_i centered i.i.d., linearizes the softmax in the small-logit regime, and shows that a simplified load-balancing loss is minimized by routers satisfying W · μ = 0. Empirically the paper measures: (a) hidden-state distance vs. logit distance on OpenWebText across five open MoEs (gpt-oss-20b, Ling-mini, ERNIE-4.5, and two others — Fig. 1 shows the predicted triangle); (b) sequence-level pooled-hidden-state cosine similarity vs. expert-frequency cosine similarity on OpenWebText + Nemotron-CC-Math intra- and cross-dataset (Fig. 5); (c) expert overlap statistics on 30 HMMT February 2025 problems × 60 reasoning models × multiple seeds via all pairwise (question, model, seed) comparisons at Top-p (Fig. 6); (d) expert-usage similarity trajectories on matched-along-one-axis prompt pairs (same-concept-different-style, plain-vs-mathematical, cross-language AIME) through prefill + generation on Ling-mini (Fig. 7); and (e) an extreme pruning protocol that keeps only the top-12 experts at bottom layers and measures NLL changes across four domains, prompt-only vs. generation (Fig. 8, Fig. 9).

Results

Triangle-shape verification of Prop. 1 holds across all five MoEs on OpenWebText [Fig. 1]; the bound is much sharper than the trivial ||W|| · ||h₁ − h₂|| baseline [Fig. 13, App.].
Load-balancing-loss suppression of shared directions is confirmed by comparing router retained energy on the top singular vector vs. expert-usage similarity across depth [Fig. 2, right column], and by direct alignment inspection of router weights against singular directions [Figs. 16, 18].
Sequence-level: intra-dataset expert-usage cosine similarity clusters at higher hidden-state similarity than cross-dataset pairs, giving a mechanistic explanation for why dataset-level “specialization” heatmaps look clean [Fig. 5].
Different reasoning models solving the same HMMT problem: ~60% Top-p expert overlap at p=[value in Fig. 6], comparable to same-model different-questions overlap and far below same-model same-question overlap. Pattern is stable across all layers and all three evaluated MoEs, and across other values of p [Fig. 25, App.].
On Ling-mini prompt-pair rollouts: all three query pairs show nearly identical expert usage during prefill; during generation, some pairs stay similar while others drop sharply — the regime is not predictable from the prompt [Fig. 7].
Extreme pruning to top-12 experts in the last two gpt-oss-20b layers: <0.05 NLL increase for prompt-only input across four topics; substantial NLL increase once the pruned model generates, both on-policy and off-policy [Fig. 8, last two columns].
Duplicating input sequences (excluding the chat template) amplifies the shared-direction component enough to force expert-usage convergence even across semantically unrelated inputs on models that don’t naturally exhibit router collapse [Fig. 10].

Why it’s interesting

This paper is the mechanistic story behind the “expert specialization is real but not always interpretable” tension already visible on the wiki. It provides the missing why for Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models‘s small-batch specialization collapse (shared-direction diversity is the missing ingredient), directly complements Mitigate Silent Expert Death In Ultra-Sparse MoE‘s “standard metrics are blind to real router behavior” finding by explaining what real router behavior is (a function of hidden-state geometry), and reframes the entire MoE Routing Design design space: since routing is a linear readout of representations, architectural changes to the router (path constraints, TopK removal, quantile balancing) can only ever manipulate how geometry is read, not what geometry is available. It also connects to Mechanistic Interpretability by adding “MoE routing patterns” to the list of activation-space phenomena that require hidden-state geometry to be understood, alongside emotion vectors, persona vectors, and Block-Sparse Featurizers.