MoRE: Mixture of Reused Experts

TL;DR

MoRE (Mixture of Reused Experts) is a hybrid between standard Mixture-of-Experts and Recurrent/Looped Transformers: groups of adjacent layers share a single enlarged pool of FFN experts, while each layer keeps its own router, attention, and norms. With R=2 and the per-layer expert count halved, the number of unique physical experts stays constant versus a DeepSeek-V3-style baseline, but each layer selects from a larger shared pool, expanding routing combinations without adding parameters. The authors report consistent language-modeling improvements over MoE and MoEUT baselines at Small/Medium/Large scales, with the gap widening at the largest expert counts, and observe that reuse concentrates in early layers — suggesting MoRE lets the model amortize generally-useful feature extractors while freeing pool capacity for specialization. Accepted to COLM 2026.

Key claims

Standard MoE architectures couple routing flexibility to memory footprint: more expert combinations per layer requires more unique experts [§1]. MoRE breaks this coupling by sharing expert pools across adjacent layers while keeping per-layer routers [§3.2].
The sharing is implemented via simple weight tying — minimal code change to existing MoE implementations — and each layer retains its own router, attention module, normalization, and a learned depth embedding to disambiguate reuse context [§3.2].
To match a DeepSeek-v3 baseline with N experts per layer, MoRE with reuse factor R=2 allocates 2N experts per shared pool so the count of unique physical experts stays constant, but each layer now routes over the larger shared pool [§3.2].
Expert reuse concentrates in early layers [Fig. heatmaps], consistent with the intuition that early layers implement broadly useful feature extraction that benefits from amortization across depth.
MoRE’s average reuse rate roughly doubles the baseline sparsity ratio: for MoRE 1.15B, K/N ≈ 5.0% but average reuse rate rises to 10.3% [§discussion].
At Small / Medium / Large scales, MoRE outperforms MoEUT, Dense, and DeepSeek MoE baselines at matched unique-physical-expert count [Table 1 part (c)], with the memory-vs-performance trade-off widening at the largest expert count (13.97 vs the DeepSeek baseline).

Method

MoRE partitions a Transformer’s layers into groups of R adjacent layers and assigns each group a single shared pool of FFN experts of size R × N (where N is the per-layer expert count in the matched standard-MoE baseline). Each layer in the group retains its own router (which produces top-K scores over the shared pool), its own attention projections and layer norms, and a learned depth embedding added to router inputs so the same expert can be routed differently at different depths. Expert sharing is implemented via weight tying across layers in a group — no changes to the MoE kernel or routing primitive are required. Load balancing follows the baseline’s standard recipe (not re-designed). The authors compare against three families at matched parameters: Dense, MoEUT (universal transformer with MoE), and DeepSeek-v3-style MoE, across Small, Medium, and Large model sizes with varying expert counts.

Results

MoRE matches or beats the DeepSeek MoE baseline at matched unique-physical-expert count across all three scales, with the gap widening as expert count grows: at the largest configuration, MoRE achieves 13.97 vs the DeepSeek baseline [Table 1 part (c)]. On MoEUT-style configurations, MoRE improves across Small/Medium/Large [Table 1]. Routing heatmaps show reuse heavily concentrated in early layers — the shared pool’s top-used experts are repeatedly selected by the first group’s layers [Fig. deepseek-expert_layer_usage_heatmap_by_depth, more_expert_layer_usage_heatmap_by_depth]. At 114M–104B active-parameter configurations spanning the compared sizes, MoRE’s effective expert pool per layer doubles (K/N = 7/139 ≈ 5.0% → average reuse rate 10.3% for MoRE 1.15B) at no extra parameter cost.

Why it’s interesting

MoRE sits exactly at the intersection of two separately-active threads on the wiki: the MoE-routing-redesign cluster (MoE Routing Design) and the parameter-sharing / recurrent-depth cluster (Looped Transformers). It is adjacent to Path-Constrained Mixture-of-Experts (PathMoE), which shares router parameters across adjacent layers to amplify natural path concentration; MoRE shares the experts themselves while keeping routers independent — the opposite constraint on the same block-of-layers structure. It is also a close cousin to STEM: Scaling Transformers with Embedding Modules (STEM), which eliminates the router entirely in favor of per-token embedding lookup: MoRE and STEM both attack MoE’s memory footprint, but MoRE preserves the full MoE routing machinery while STEM replaces it. The early-layer-reuse finding converges with ELT: Elastic Looped Transformers for Visual Generation (ELT) and Hyperloop Transformers (Hyperloop)‘s “middle cycle” layout intuition — both argue that intermediate / early blocks are the natural locus of weight sharing — and offers a candidate mechanistic answer to the open question raised by The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertise about where in a MoE expert specialization localizes.