Routing-Free Mixture-of-Experts

TL;DR

A bottom-up MoE redesign from LMU Munich (Liu, Han, Yan, Tresp, Ma) that eliminates all centralized routing machinery — external router, Softmax, TopK, and hard-coded load-balancing target — and lets each expert decide its own activation from its own internal score plus a learnable bias. Builds on AoE (expert-internal scoring) and ReMoE (ReLU gating without TopK/Softmax) and combines them: AoE’s low-rank internal norm feeds a ReMoE-style ReLU gate with per-expert bias, giving a fully decentralized self-activation rule. Pairs this with a unified load-balancing auxiliary loss that interpolates between expert-balancing and token-balancing via a single mixing coefficient $\beta$ , plus an adaptive coefficient that drives empirical activation density toward a target. At scales up to 0.8B on OpenWebText, beats standard MoE, AoE, and ReMoE on perplexity and 9-benchmark zero-shot average; gains do not diminish with scale, and the model tolerates a wider learning-rate range before training collapses.

Key claims

Standard MoE imposes three rigid inductive biases that are not theoretically motivated: an external router with tiny capacity learning expert preferences via indirect trial-and-error, a hard TopK that enforces uniform sparsity regardless of input complexity, and a Softmax that discards absolute activation magnitudes by forcing competitive normalization [§1, §2].
AoE (Lv et al. 2025) addresses the router-capacity bottleneck by scoring each expert from its own internal low-rank projection $\|W^{down}W^{up}x\|$ but still feeds that score through centralized TopK + Softmax — retaining the inductive biases [§3.1, Eq. 6-7].
ReMoE (Wang et al. 2024b) addresses TopK + Softmax by replacing them with a single ReLU on the router output, preserving magnitude information and yielding natural sparsity — but still keeps a centralized external router [§3.1, Eq. 8].
Routing-Free MoE combines AoE’s internal scoring with ReMoE’s ReLU gating and adds a learnable per-expert bias before the ReLU, so each expert activates iff $\text{ReLU}(\|W^{down}W^{up}x\| + b_i) > \alpha$ for a global threshold $\alpha$ — fully decentralized, no router, no TopK, no Softmax [§3.1, Eq. 9-10].
The non-differentiable binary activation gate is trained via a differentiable pre-threshold proxy (the gate value itself) inside the auxiliary loss; this is the same trick ReMoE uses for ReLU gating [§3.2, Eq. 11-12].
Load-balancing is unified: expert-balancing loss $\mathcal{L}_{EB}$ penalizes per-expert load deviation; token-balancing loss $\mathcal{L}_{TB}$ penalizes per-token activation-count deviation; the two are combined as $\mathcal{L}_{LB} = (1-\beta)\mathcal{L}_{EB} + \beta\mathcal{L}_{TB}$ with $\beta \in [0,1]$ a single configurable knob [§3.2, Eq. 13-15].
The auxiliary-loss coefficient $\alpha_{LB}$ is adaptive — at each step it grows when empirical density exceeds the target and shrinks when below, removing the need to manually tune the LB weight [§3.2, Eq. 17].
Per-expert bias initialized to $+\infty$ (all experts active) lets sparsity emerge gradually from a fully participatory warm-up phase, avoiding expert collapse [§3.2].
At scale S (~93M params, 91M FLOPs), Routing-Free MoE achieves 27.42 perplexity vs. standard MoE’s 31.22, AoE’s 30.00, and ReMoE’s 29.60 — a 12% drop over standard MoE on iso-FLOPs [Table 1].
Gains hold at M (~290M) and L (~810M): standard MoE 25.00 / 24.58 PPL, Routing-Free MoE 22.08 / 19.97 PPL — the advantage grows with scale rather than diminishing [Table 1, Fig. 2].
Zero-shot average over 9 benchmarks (PIQA, HellaSwag, WinoGrande, ARC-e/c, OpenBookQA, QQP, QNLI, SST-2) is higher at every scale: S 39.77 vs 38.96, M 40.40 vs 39.64, L 40.76 vs 40.00 [Table 1].
Training-stability LR sweep: standard MoE collapses at higher learning rates while Routing-Free MoE remains stable across a wider range; at scale L, tuning the LR for standard MoE yields only marginal gains over its scale-M number, whereas Routing-Free MoE continues improving with scale [§4.4, Fig. 5].
Per-layer vs. global density enforcement matters: enforcing the activation target $\rho$ globally (across all layers) rather than per-layer drops scale-S perplexity from 39.44 to 28.74 — the per-layer-uniform-sparsity inductive bias is harmful, and the model self-organizes into a non-uniform depth-wise activation pattern when freed [§5.1, Fig. 6, Appendix D.2].
Token-balancing and expert-balancing are complementary, not mutually exclusive: the balanced setting $\beta=0.5$ achieves the lowest perplexity (28.34) and the highest eval throughput (662.3) — both pure TB ( $\beta=0$ , 28.41 PPL) and pure EB ( $\beta=1$ , 28.43 PPL) underperform [Table 4, Fig. 7].

Method

The architecture is a Mixtral-style decoder (GQA + RoPE + SwiGLU + RMSNorm) where each MoE layer is replaced with $N$ Routing-Free experts in parallel. Each expert $i$ owns a low-rank “internal scorer” — two small matrices $W^{down}_i \in \mathbb{R}^{r \times d}$ , $W^{up}_i \in \mathbb{R}^{d \times r}$ — plus the standard SwiGLU FFN. For input hidden state $x$ , expert $i$ computes its internal score as $s_i = \|W^{down}_i W^{up}_i x\|$ , adds a learnable scalar bias $b_i$ , applies a ReLU, and compares against a global threshold $\alpha$ : if $\text{ReLU}(s_i + b_i) > \alpha$ , the expert activates and contributes $\text{ReLU}(s_i + b_i) \cdot \text{FFN}_i(x)$ to the layer output; otherwise it contributes zero. There is no router, no TopK selection, no Softmax normalization across experts, and no information channel between experts at activation time.

Training balances three pressures via the auxiliary loss. The target activation density $\rho$ (analog of $k/N$ in TopK) is enforced only via the adaptive $\alpha_{LB}$ feedback loop on the global density across all layers — per-layer densities are free to deviate. The load-balancing loss $\mathcal{L}_{LB}$ combines (1) expert-balancing (per-expert load uniformity across tokens) and (2) token-balancing (per-token activation-count uniformity), interpolated via $\beta$ . Both losses use the same trick: a binary non-differentiable indicator times the differentiable gate value as a proxy, so gradients still flow through the gate even though the on/off decision doesn’t. Initialization sets each $b_i$ to $+\infty$ so every expert starts active; as $\alpha_{LB}$ climbs during warm-up, biases pull experts below the threshold and sparsity emerges.

The closest predecessors are AoE (which kept TopK + Softmax) and ReMoE (which kept the router); Routing-Free MoE is the natural composition of both ablations plus the per-expert bias that lets ReLU gating work on a strictly-non-negative internal score.

Results

Iso-FLOPs perplexity, scale S (93M params, 91M FLOPs): Standard MoE 31.22 → AoE 30.00 → ReMoE 29.60 → Routing-Free MoE 27.42 [Table 1].
Scaling: at M (290M / 248M FLOPs) RFMoE 22.08 vs MoE 25.00 (−11.7%); at L (810M / 608M FLOPs) RFMoE 19.97 vs MoE 24.58 (−18.8%). The gap widens with scale [Table 1, Fig. 2].
9-benchmark zero-shot average: scale L Routing-Free MoE 40.76 vs MoE 40.00 — modest but consistent across all 9 tasks [Table 1].
Architectural ablation at scale S (Table 2): going from MoE (31.22) to AoE-only (30.00, $r=16$ ) to ReMoE-only (29.60) to RFMoE- $r$ =16 (28.73) to RFMoE- $r$ =32 (28.33). Each component contributes; the combination is super-additive.
Internal-rank $r$ ablation (Table 3): $r=8$ → 29.16, $r=16$ → 28.74, $r=32$ → 28.34, $r=64$ → 28.24. Default $r=32$ ; diminishing returns past that.
LR-sweep stability (Fig. 5): at scale S, standard MoE diverges at $\eta = 1e^{-3}$ while RFMoE trains stably; at scale L the RFMoE advantage grows under LR tuning while standard MoE saturates.
Per-layer vs. global density (Appendix D.2): per-layer 39.44 PPL vs. global 28.74 PPL — an 11-point swing. The trained model self-organizes activation density to peak in mid-depth layers and fall toward both ends.
Balancing-mix $\beta$ sweep (Table 4): $\beta=0.5$ wins on both perplexity (28.34) and throughput (662.3 tok/s), with pure-EB and pure-TB each ~0.07-0.09 PPL worse.

Why it’s interesting

This is the paper that closes a loop the wiki has been tracking: the “remove a design knob from MoE” thread that runs through DynaMoE: Dynamic Token-Level Expert Activation with Layer-Wise Adaptive Capacity for Mixture-of-Experts Neural Networks (remove fixed TopK via percentile gating, sweep schedules), Path-Constrained Mixture-of-Experts (remove per-layer-router-parameter axis by sharing weights), Quantile Balancing: A Hyperparameter-Free MoE Load Balancing Method (remove load-balancing-loss tuning), and Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design (give an explicit scaling law for the attention-vs-experts FLOPs split). Routing-Free MoE is the most aggressive of the set — it removes the router, TopK, and Softmax simultaneously — and it ships scaling evidence (gains widening at 0.8B) and a stability story (wider LR tolerance) rather than just a toy-scale demo. Pair this with the empirical finding that global rather than per-layer density enforcement gives an 11-point PPL swing, and the depth-wise activation pattern that emerges — and the message is consistent with PathMoE’s “the natural expert-path distribution is non-uniform, stop fighting it.” This is the fourth MoE-architecture paper in the cluster, enough to spin up a dedicated moe-routing-design concept page; that page is created in this same PR.

Two caveats worth flagging for Luma reading. (1) The biggest scale is 0.8B and the training corpus is OpenWebText for one epoch — this is a strong proof-of-concept, not a guarantee that the per-expert bias trick and the adaptive $\alpha_{LB}$ feedback don’t develop pathologies at production scale. (2) The model is trained from scratch; adapting a pretrained standard MoE into Routing-Free MoE is explicitly listed as future work and the authors don’t claim a cheap conversion recipe. The “flexible activation pattern” claim is genuinely supported by the per-layer density visualizations and the LR-tolerance sweep, but it needs replication at frontier scale before becoming a design default.