Path-Constrained Mixture-of-Experts

TL;DR

Apple paper (Gu, Likhomanenko, Thilak, Ramapuram, Jaitly) that reframes MoE computation through the lens of expert paths — the sequence of expert selections a token makes across all $L$ layers. Despite $N^L$ possible paths over $N$ experts and $L$ layers, tokens in practice cluster into a tiny fraction that align with linguistic function; the rest of path space is wasted. PathMoE exploits this by sharing router parameters across blocks of consecutive layers — a middle point on a spectrum between fully independent per-layer routers (standard MoE) and a single shared router for all layers. Analysis claims the constraint amplifies the natural path concentration: more concentrated path clusters, better cross-layer routing consistency, and greater robustness to routing perturbations. This is the architectural counterpart to the same group’s earlier Omni-router work on ASR, which shared a single router across all layers.

Key claims

Sparse MoEs route each token independently at every layer, leaving $N^L$ possible expert paths; in practice tokens occupy a small fraction of these paths, with the occupied subset correlating with linguistic function — a statistical inefficiency [§1, Abstract].
Constraining the effective path space can amplify this natural concentration rather than fight it; PathMoE instantiates this by having all layers within a contiguous block share the same router parameters [§1, Fig. 2].
The design sits on an explicit spectrum: (a) independent per-layer routers (standard MoE), (b) block-wise parameter sharing (PathMoE), (c) fully shared decision (one router, one routing choice for all layers) [Fig. 2].
Motivation for the sharing assumption is that residual connections make adjacent layers’ input distributions similar, so similar (but not identical) routing is reasonable to share [§1, follow-up to Omni-Router reasoning].
PathMoE produces more concentrated path clusters than independent routing [Abstract, analysis section].
PathMoE shows better cross-layer routing consistency [Abstract].
PathMoE is more robust to routing perturbations than independent routing [Abstract].

Method

The PathMoE recipe modifies a standard sparse-MoE transformer in one place: instead of every MoE layer owning its own router projection, contiguous layers are grouped into blocks and all layers within a block share router parameters. Layer $\ell$ in block $b$ still does its own top- $k$ selection over its own expert set (experts are not shared — only the router that scores them), but because the router weights are identical the selection logits in early-block layers are highly correlated with later-block layers’ logits, differing only by what residual updates have done to the input representation. This constrains the reachable path space without collapsing it to a single universal path (which would be (c) on the Fig. 2 spectrum and would not let layers diverge as the residual stream specializes).

Analytically the paper studies the empirical path distribution under (a) and (b): how many distinct paths a corpus of tokens occupies, how concentrated those occupied paths are (entropy / Gini-like metric), and how stable the path assignment is under small input perturbations. The Apple group’s earlier Omni-Router paper (Gu et al. 2507.05724) is the ASR-domain ancestor of the same idea, taken there to its full-sharing extreme (one router for all layers).

Results

The paper’s experimental claims (full sweep numbers in the PDF; this filing relies on the abstract and HTML preview because the arxiv endpoint was rate-limiting at filing time):

More concentrated path clusters under PathMoE vs. independent per-layer routing — measured on the empirical token-path distribution [Abstract].
Better cross-layer consistency in routing decisions [Abstract].
Greater robustness to routing perturbations (e.g. small input noise should not flip the entire path) [Abstract].
The result is framed as an architectural lever — orthogonal to load-balancing-loss design, expert-count schedules, or attention/FFN FLOPs allocation.

Why it’s interesting

Kyle’s note that “someone at Luma suggested doing this in the last few weeks” is the relevant signal — Luma researchers independently arrived at router-sharing as a natural compression of the MoE design space, and this is the public paper that says it works. Connects cleanly to three already-filed MoE pieces. (1) PathMoE is complementary to Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design: that paper gives a scaling law for how much compute to allocate between attention and experts; PathMoE constrains how the router itself is parameterized. The two axes don’t conflict — you’d want both. (2) It contrasts with DynaMoE: Dynamic Token-Level Expert Activation with Layer-Wise Adaptive Capacity for Mixture-of-Experts Neural Networks, which sweeps per-layer expert counts and active-set sizes at toy scale and finds the optimal schedule is task-dependent — PathMoE goes the other direction and shares router parameters across layers rather than diversifying them. (3) It sits adjacent to Quantile Balancing: A Hyperparameter-Free MoE Load Balancing Method and A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training under the same broader theme of “remove a design knob from MoE”: Su’s paper removes the load-balancing-loss tuning axis; PathMoE removes the per-layer-router parameter axis. The Omni-Router ASR predecessor (Gu et al. 2025) suggests this is a real, repeatedly-rediscovered pattern — and there are now enough MoE-routing papers on the wiki to think about a dedicated moe-routing-design concept page once one more lands.