Multi-View Latent MoE: Letting Every Router Shape Expert Computation

TL;DR

Multi-View Latent MoE attacks the MoE routing problem from a new angle: a single router’s expert ranking only weakly agrees with dense-eval expert-utility rankings, especially in middle and deep layers. Rather than redesign the router architecture or load balancing, this work runs multiple independently-initialized routers in parallel and aggregates their sigmoid scores to pick the top-k experts. A second step — “Multi-View” — reuses each selected expert’s relative scores across routers as mixing weights over learned latent views of the expert’s input, so one expert still does one forward pass but gets a per-router-shaped input. On a 10B-total / 1B-active MoE trained on 708B tokens, the full method adds +1.18 pp on a nine-task downstream mean; the Multi-Router piece alone gives +0.6 pp and improves router-vs-utility rank correlation by ~58% in the worst layer.

Key claims

Single-router expert rankings only weakly agree with dense-evaluation utility rankings, and the misalignment concentrates in middle and deep layers [§Abstract].
Multi-Router aggregates sigmoid scores from several independently-initialized routers; after training the routers develop differentiated preferences so experts strongly favored by one router or jointly supported by several get selected [§Abstract].
Multi-View uses each selected expert’s relative scores across the routers as mixing weights over learned latent views of the expert’s input, so the same expert gets a per-router-informed input without being executed multiple times [§Abstract].
Each selected expert still runs exactly one forward pass, preserving active-expert count, expert FLOPs, and all-to-all communication volume [§Abstract].
At 24 layers / 10B-total / 1B-active / 708B tokens, full Multi-View Latent MoE improves the nine-task downstream mean by 1.18 pp over the baseline [§Abstract].
Multi-Router alone contributes 0.6 pp of the gain and boosts router–utility rank correlation by 8.1% globally and ~58% in the layer with the worst single-router alignment [§Abstract].
Controlled mixing interventions confirm the model actually uses input differences that depend on both expert identity and token — i.e., the latent-view mixing is a real signal path, not noise [§Abstract].

Method

The baseline is a standard sparse MoE layer: a single linear router scores all experts, top-k selection picks the active set, and each selected expert runs its FFN on the same token representation. Multi-View Latent MoE replaces this in two stages.

Multi-Router keeps the fixed active-expert budget but replaces the single router with several parallel routers, each a linear projection initialized independently. Per expert, the sigmoid scores from the routers are aggregated into a single selection score; top-k is applied to the aggregate. Independent initialization is what lets the routers drift into differentiated preferences during training — experts picked up by any strong router preference survive, as do experts that are co-supported across routers, which is the mechanism that pulls up the router-vs-utility rank correlation.

Multi-View then reuses the information already computed in the Multi-Router stage. For each selected expert, its vector of per-router scores (normalized to relative weights) is used to mix a small set of learned latent views of the token into that expert’s input. The expert itself still executes one forward pass; the per-router preference structure is encoded into the input each expert sees. The invariants are explicit: active-expert count, per-expert FLOPs, and all-to-all dispatch volume all match the single-router baseline, so the added cost sits in the (small) router projections and the latent-view mixing.

Results

Backbone: 24-layer, 10B total / 1B active MoE trained on 708B tokens [§Abstract].
Nine-task downstream mean: +1.18 pp over the single-router baseline for full Multi-View Latent MoE; +0.6 pp for Multi-Router alone [§Abstract].
Router–utility rank correlation (Multi-Router): +8.1% globally and ~58% relative improvement in the layer whose single-router alignment was weakest [§Abstract].
Controlled mixing interventions on the latent views show the model’s output depends on both expert identity and token identity — evidence the view-mixing is being used, not bypassed [§Abstract].

Why it’s interesting

This work opens a new axis on the MoE-routing design board tracked in MoE Routing Design: rather than fighting the router (replace TopK, remove Softmax, remove the router entirely as in Routing-Free Mixture-of-Experts / STEM: Scaling Transformers with Embedding Modules), or constrain router parameters across layers (Path-Constrained Mixture-of-Experts), or rescope the load-balancing signal (Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models), it adds routers and uses them for both selection and per-expert input shaping. Also distinct from The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertise, which argues the router is a linear read of hidden-state geometry so richer router capacity should matter; Multi-View Latent MoE is a concrete instantiation of that prediction at 10B scale while holding the FLOPs invariants that make the lever deployable. Explicitly validates the misalignment at depth that single-router MoE recipes silently accept.