Skip to content
Bud's Wiki

MoE Routing Design

TL;DR

Section titled “TL;DR”

A cluster of 2026 papers questioning the rigid inductive biases of the standard Mixture-of-Experts routing pipeline — external router, TopK, Softmax, hard-coded per-layer expert counts, fixed attention/FFN FLOPs split, auxiliary load-balancing loss with a tuned coefficient. Each contribution removes or replaces one (or more) of these design axes: the architecture-side end ranges from share router weights across layers (PathMoE) to eliminate the router entirely (Routing-Free MoE, STEM); the training-side end ranges from unify token-balancing and expert-balancing to make the load-balancing coefficient adaptive to give an explicit scaling law for attention vs. expert FLOPs allocation. Common thread: the empirical token-path distribution is highly non-uniform, and constraints that fight that non-uniformity (uniform per-layer sparsity, fixed TopK, equal-FLOPs-per-token) leave loss on the table.

Key claims

Section titled “Key claims”

Recent contributions

Section titled “Recent contributions”
  • [2026-11-06] J++ Lens: Jacobian Filtering enables more faithful workspace readouts: Suggestive counterpart to The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertise from the dense side: the J++ lens fits k-means clusters over residual-stream activations and computes per-cluster “Expert Jacobians” of the unembedding-composed-with-the-rest-of-the-network, then learns weights that filter out bad experts. The structure is visibly MoE-like — some experts generalize outside their own cluster, others give poor readouts everywhere — but the model under study is dense (Qwen3.6-27B, Gemma 4 31B, Olmo 3 32B, dense Qwen3.5-9B) in addition to MoE (Qwen3.5-122B-A10B, DeepSeek-V4-Flash). Reads as evidence that the hidden-state geometry The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertise argued drives trained MoE routing is already rich enough in dense models for k-means + per-cluster linear maps to carve functional specialization.
  • [2026-11-05] The Ultimate MoE Training Handbook: training Mixture-of-Experts models with Megatron-LM, setting by setting: Nouamane Tazi (Hugging Face) publishes an end-to-end MoE training handbook structured around Megatron-LM, with Chapter 3 enumerating router / load balancing / expert block layout as the three inside-the-layer design axes and giving recommended settings referencing DeepSeek-V3, Kimi-K2, Qwen3-Next, and GPT-OSS. Recipe-side companion to the design-space papers tracked here; fills the “if you trained dense before, do these things” gap that individual design-choice papers leave open.
  • [2026-11-05] Multi-View Latent MoE: Letting Every Router Shape Expert Computation: Multi-View Latent MoE adds a new axis to the routing-design board — router multiplicity. Diagnoses that single-router rankings weakly agree with dense-eval expert utility, especially in middle/deep layers. Runs multiple independently-initialized routers in parallel and aggregates sigmoid scores for top-k selection; then reuses each expert’s per-router score vector as mixing weights over learned latent views of the input, so each selected expert still runs exactly one forward pass (active-expert count, per-expert FLOPs, and all-to-all volume unchanged). At 10B-total / 1B-active / 708B tokens: +1.18 pp on nine-task downstream mean (full method); +0.6 pp from Multi-Router alone, with router–utility rank correlation up 8.1% globally and ~58% in the worst-aligned layer. Concrete scale-10B evidence for the Hayou et al. (The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertise) prediction that richer router reads of hidden-state geometry should help, delivered under FLOPs invariants that make it a deployable lever rather than a capacity-trade.
  • [2026-11-02] MoRE: Mixture of Reused Experts: MoRE opens a cross-depth expert sharing axis on the routing-design board: groups of adjacent layers share one enlarged expert pool while each layer keeps its own router + attention + norms + depth embedding. With R=2 and halved per-layer N, unique physical experts match a DeepSeek-v3 baseline but each layer routes over 2× the pool, expanding routing combinations without adding parameters. Reuse concentrates in early layers (reuse rate 10.3% vs K/N≈5.0% for MoRE 1.15B), consistent with the hypothesis that early layers implement broadly-useful features that benefit from amortization. Beats Dense, MoEUT, and DeepSeek-MoE at matched unique-expert count across Small/Medium/Large, with the gap widening at the largest expert count (13.97 vs DeepSeek). Complementary to PathMoE (shares routers across adjacent layers, not experts) and STEM (eliminates routing entirely); sits adjacent to the Looped Transformer family on the parameter-sharing axis. COLM 2026.
  • [2026-10-26] Agentic World Action Models: AW-0 is a three-expert Mixture-of-Transformers where modality routing is explicitly asymmetric by construction rather than learned: a code expert (LLaDA diffusion LM + frozen Qwen3-VL ViT, discrete-token absorbing-mask denoising), a video expert (pretrained Cosmos-Predict2.5 DiT, continuous VAE-latent flow matching), and an action expert (separate DiT, continuous robot-state-conditioned flow matching). Each expert keeps its own attention projections, LayerNorms, and FFNs; cross-expert communication happens at every layer through masked joint attention with bidirectional attention inside each expert and blockwise causal attention across experts in the order code → video → action. Controlled ablation on the attention ordering (C → V → A vs V → C → A) finds that placing the code expert first wins on 7/8 real-world bimanual tasks (71.1 vs lower, no exact V→C→A number tabulated) — first filed MoT instance where expert ordering is the ablation lever rather than router topology. ViT-vs-VAE ablation on the code expert’s visual input (keeping the backbone fixed) adds a per-expert input-adapter lever to the MoT design space: Qwen3-VL ViT beats reusing the video expert’s Wan2.2 VAE features on all 8 real-world tasks (71.1 vs 55.6 avg, +25 pp on Pack Backpack, +18 pp on Solve Cube) because VAE features are built for pixel reconstruction and under-support the semantic distinctions needed to select and compose skills. Complements the LingBot-VA / H3-World / Rho-1 / Flex-π MoT lineage with the first filed three-expert instance where one expert operates on discrete tokens with absorbing-mask denoising while the other two operate on continuous latents with flow matching.
  • [2026-10-16] Introducing Muse Spark Open — The Most Capable US Open-Weight Model to Date: Muse Spark Open publishes an 816B-total / 42B-active sparsity ratio (~19× sparsity), landing in the same scale class as Kimi K3 (2.8T total) and DeepSeek-V4-Pro but with a tighter active-parameter envelope. Blog-level disclosure gives no expert-count / routing-algorithm / shared-expert details — those are deferred to the model card and a forthcoming evaluation report. Added as a scale-ratio datapoint only; architectural detail flagged for follow-up when the model card is fetched.
  • [2026-10-15] Frontier Pre-training at Microsoft AI (MAI-Base-2): MAI-Base-2 (Microsoft AI, 2.8T/80B MoE) ships a dropless multi-round MoE with tunable fixed per-expert memory capacity: experts grouped per EP rank run multiple dispatch → compute → collect rounds at fixed capacity, overflow tokens use unused capacity from other experts in the same group before spilling to extra rounds, backward recompute runs per-expert-group, and all phases are pipelined so all but the first dispatch and last collect overlap with expert compute. Removes the memory-fragmentation + rare-imbalance-dominates-allocation failure mode that pushed MAI-Thinking-1 to interleaved MoE-every-other-layer; combined with device-resident token metadata, GPU-initiated all-to-all, symmetric-memory all-to-all over NVLink without staging copies, and batched LBL all-reduce across layers, MoE-layer MFU rises 24% → 35% and MoE-in-every-layer flips to being EGtime-optimal. Concrete frontier-scale instance (16K GB200 GPUs) that the systems bottleneck — not the FLOPs trade-off — was what gated MoE-every-layer in the prior interleaved generation.
  • [2026-10-13] The Will to Act — Natura-1: Natura-1 adds a 50B-param MoE applied to a VLA foundation model datapoint distinct from the LLM-pretraining-focused MoE recipes dominating this page — Nature Will’s blog frames the design as kept close to standard mixture-of-experts transformer, with algorithmic innovations in the arrangement of multimodal tokens within a single sequence and the causal structure of attention rather than in router design, so the backbone scales further. Load-bearing per the authors is holding the MoE architecture simple and standard while the signal lies in sequence structure + attention causality for unifying language + vision + world modeling + action generation. First MoE VLA at the 50B / 1M-hour-egocentric-pretraining scale on file; expert count / activation sparsity / routing-loss details not disclosed in the public blog.
  • [2026-10-01] ID Balancing: Stable Training of Extremely Sparse MoE via PID-Based Load Control: Unifies DeepSeek’s loss-free bias update (fixed-step integral controller) and Kimi K3’s Quantile Balancing (generalized proportional controller) as endpoints of a PID family, then fills in the middle with ID Balancing — an Integral-Derivative controller that scales the integral term with load error and fires the derivative term only when imbalance is worsening. At 768-expert routing with Top-3, cuts worst-case backbone MaxVio by >50% and training-average MinVio by >12% vs best baselines; scaling 18.9B → 69.9B at Top-10-of-768 leaves worst-case MaxVio nearly flat and ~89.6% below the auxiliary-loss baseline. Advantages grow monotonically with sparsity, extending the aux-free frontier past Kimi K3’s Top-16-of-896 recipe.
  • [2026-09-22] MiMo-V2.6: Scaling Reinforcement Learning Towards Self-Improvement: MiMo-V2.6 freezes the MoE router during RL as a stabilizer for scaled ultra-sparse MoE training (Pro: 384 experts, top-8, no shared experts; Flash: 256 experts, top-8) — a distinct position from expert-death-mitigation approaches (Mitigate Silent Expert Death In Ultra-Sparse MoE) that argues router drift under RL, not expert death, is the killer at scale. Combined with a multi-layer reward-hacking defense and Muown (Muon variant) for hidden weights while router stays on AdamW.
  • [2026-09-13] The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertise: Hayou et al. give the mechanistic story behind expert specialization: because the router is a linear map from hidden states to expert logits, hidden-state similarity is both necessary and sufficient for expert-usage similarity (Prop. 1, tight data-geometry-aware upper bound verified across 5 pre-trained MoEs). Prop. 2 formalizes why load-balancing loss suppresses shared hidden-state directions to maintain routing diversity, providing the theoretical mechanism behind the small-batch specialization collapse Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models observed empirically — and predicts that auxiliary-loss-free models (Moonlight 16B, Trinity Mini) should not show the same shared-direction suppression (confirmed). Three stress tests then show specialization patterns still resist human interpretation despite the clean mechanism: different reasoning models solving the same HMMT problem show only ~60% expert overlap (comparable to same-model different-questions); prefill expert usage is not predictive of generation-phase usage; and in deep layers of gpt-oss-20b, semantically unrelated prompts collapse to identical expert activation during prefill but diverge sharply during generation. Reframes the entire routing-design board: architectural changes only manipulate how geometry is read, not what geometry is available.
  • [2026-09-12] Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data: Jha, Li, Leskovec, Liang & Zettlemoyer add a sparsity-conditioned overfitting axis to the routing-design board: at matched active params 80M–1B, MoEs degrade under data repetition strictly faster than dense (dense tolerates 8× minimally; MoEs start suffering at 4× and drop below dense after 32×), with degradation scaling with total rather than active params. Mechanism ties directly to the “training metrics are blind to real router behavior” thread — router decisions stabilize very early and post-stabilization expert-specialization degree predicts overfitting severity. Strong dropout can push MoE back ahead past 64× but no method fully recovers all-unique-data performance. Sharpens the open question about specialization-as-diagnostic that PathMoE and Demons-in-the-Detail already raised.
  • [2026-09-10] DeepSeek-V4.1-Flash — Pushing the Limits of KV Cache Compression: DeepSeek-V4.1-Flash keeps the DeepSeek-V3 MoE routing recipe (1 shared + 384 routed experts, top-6 per token, auxfree-bias balancing) but pairs it with two axes new to this page’s design-space board: (a) the Causal Encoder-Decoder backbone makes activated parameter count asymmetric between prefill (8B) and decode (16B) — the MoE budget shifts phase-dependently rather than being a static per-token number, and (b) a separate 196B-parameter Engram conditional memory bank sits outside the MoE budget and is accessed via sparse token-based lookup, making V4.1-Flash’s 552B total a composition of {backbone + MoE experts} + parametric memory rather than a monolithic sum.
  • [2026-09-03] Introducing K2 Horizon: Frontier Performance, Radically Open: K2 Horizon 36B-A4B introduces MoVA (Mixture-of-Value Attention) — a genuinely new axis on the routing-design board: sparse expert routing inside the attention value component of multi-head attention, rather than in the FFN. Claimed compatible with FlashAttention, GQA, and sparse attention. Paired with MoE FFN layers; activates ~4B of 36B parameters per token and (under matched training on the same 22T tokens as the dense 32B) performs “only slightly below” the dense sibling. Complementary to prior FFN-side directions — PathMoE (layer-shared router), Routing-Free MoE (no router), STEM (embedding-module replacement) — and to MAGI-2’s head-parallel FFN routing. First production-scale open-weights instantiation of routing-inside-attention on the tracker.
  • [2026-08-27] Liam Fedus reflects on Switch Transformers — 1.6T-param / 2048-expert MoE was ahead on perplexity, behind on reasoning: Liam Fedus retrospective on Switch Transformer (2020): 1-of-2048-expert routing at <3B active / 1.6T total beat T5 on C4 perplexity and set TriviaQA SOTA but failed on SuperGLUE-style reasoning — historical anchor that the current 2026 return to ultra-sparse MoE (Kimi K3 16-of-896, MAGI-2 12×256, Qwen3-Next 512-expert) is a re-litigation of a 2020 bet that originally lost on reasoning tasks.
  • [2026-08-26] Isaac 0.5: An Open-Weight Embodied Foundation Model: Isaac 0.5 deploys null-expert routing at 36B production scale: each of 40 backbone blocks has 256 real routed MLPs plus one learned null-router row that is duplicated into 256 tied null copies and concatenated with the 256 real-expert scores; top-8 over the 512-score pool selects kreal = min(8, j) real experts where j is the count of real scores strictly above the shared null score, so each token executes 0–8 routed MLPs while the full 256-expert library remains available. The 256-copy multiplicity matters through the softmax partition function — it contributes 256·exp(u∅), equivalent to a single null score with logit u∅ + log 256, shifting the null-path initial probability mass without adding parameters. At the 50% null reference (4 of 8 routes real) active params are 2.5B of 36B, and MFU is 24% in the hyper-sparse released config vs 48% dense. Concrete deployed instance of the token-adaptive routed-compute direction on this page’s design-space board — closer in spirit to Routing-Free Mixture-of-Experts and DynaMoE: Dynamic Token-Level Expert Activation with Layer-Wise Adaptive Capacity for Mixture-of-Experts Neural Networks than to fixed-topK DeepSeek-style routing, with a clean parameter-free knob (null multiplicity) for the initial null-path mass.
  • [2026-08-26] Qwen3.8-Flash-Next — Hybrid Attention with QSA, Gated Residual, N-gram Embedding: Qwen3.8-Flash-Next inherits the 512-expert / 10 routed + 1 shared configuration from Qwen3-Next-80B-A3B but at 125B total / 6B active, and adds a 51B N-gram Embedding table as an alternative parameter-scaling axis alongside MoE — first filed model where architectural parametric memory co-exists with an ultra-sparse MoE as two distinct scaling levers.
  • [2026-08-25] Mitigate Silent Expert Death In Ultra-Sparse MoE: Documents silent expert death in ultra-sparse MoE (top-8-of-768 at 60B/180B): lower-layer routed-expert norms fall to ~1/10 of init, masking them barely changes model quality, yet training loss and load-balance metrics all look healthy. Introduces LLAL (LM Loss as Auxiliary Loss) applied to an early MoE layer during the first ~3% of training — at 180B / 135k steps, 4k LLAL steps yield +2.8 MMLU and +4.8 MMLU-Pro with better validation loss and load balance. Also audits open MoEs (MiMo-V2.5-Pro, Qwen3.5-397B-A17B, Qwen3.8-Max, Kimi-K3) and reports the same collapse signature, plus Kimi-K3’s aux-free bias overriding router top-1 on >70% of tokens in some layers — sharpens the “standard metrics are blind to real router behavior” thread that Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models and Quantile Balancing: A Hyperparameter-Free MoE Load Balancing Method each attack from a different angle.
  • [2026-08-20] Marin scaling laws simulate the entire training trajectory of a 67B MoE run within 1%: Marin scales its Delphi + Quantile Balancing MoE recipe to a 67B-parameter production run (target loss hit within 1%) and to a 535B-parameter run in flight — production-scale evidence that the Quantile Balancing: A Hyperparameter-Free MoE Load Balancing Method hyperparameter-free load balancer holds beyond the 32B-A5B validation reported in that post.
  • [2026-08-19] Ornith-1.5: From Self-Scaffolding to Self-Improvement: Ornith-1.5-35B-A3B (3B active / 35B total) outperforms dense Gemma-4-31B and Muse-Glimmer-30B by wide margins on Terminal-Bench 2.1 (Claude Code) 68.5 vs 43.4 / 51.7 and SWE-Bench Verified 79.0 vs 52.0 / 76.0 — another datapoint that low-active-ratio MoE beats dense at fixed parameter count on agentic-coding tasks when paired with sufficient post-training.
  • [2026-08-14] Qwen3.8 open weights release — 27B dense multimodal + 2.4T-A95B MoE (Alibaba Qwen): Qwen3.8-2.4T-A95B — 2.4T-total / 95B-active MoE open weights (Apache 2.0) — extends the 2026 open-weights MoE frontier cohort alongside Kimi K3 (2.8T total) and Nemotron 3 Ultra (550B/55B-active). The active-parameter fraction (~4%) sits between Kimi K3’s ultra-sparse (16-of-896) and Nemotron 3 Ultra’s denser 10% ratio.
  • [2026-08-12] MAGI-2 Preview: Scaling Video Generation Models Efficiently: Sand.ai MAGI-2 Preview pushes MoE granularity to the extreme end of the cluster: 12 independent 256-d routing heads per layer, each with its own router and pool of 256 experts, top-6 per head — 3,072 head-local expert units per layer, 72 activated per token, at 114B total / 6B active. Motivation is that long video sequences make standard token-choice MoE’s dynamic all-to-all a first-order systems cost; Head Parallel regularizes cross-node payloads to fixed head-slice shapes. Load balancing is auxiliary-loss-free bias applied independently per routing head. Concrete architectural answer to the “does eliminating routing inductive biases transfer past 1B” open question — MAGI-2 goes the opposite direction (more, finer routers) at 114B.
  • [2026-08-07] Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes: Han et al. arrive at the decouple-FFNs / share-attention-and-norm recipe from the component-sharing side — a five-way ablation (fully shared dense, FFN-only decoupled, FFN+Attn decoupled, FFN+FinalNorm decoupled, fully split) shows fully shared blocks maximize competition and fully split blocks lose synergy, with FFN-only decoupling as the best of the five. Converges with ERNIE 4.5’s modality-isolated-router + shared-attention finding from the routing-design direction. The result holds across RAE, Raw Pixels, CLIP+VAE, and AR UniTok visual designs (Raw Pixels gives the largest language gain ΔPPL −0.266). At 13.5B-total / 1.5B-active MoE / 2T tokens, the recipe with L70/U25/G5 asymmetric mixing beats a dense 3.5B baseline on every axis.
  • [2026-07-31] Chimera: Designing and Chinchilla-Scaling Hybrid Visual Diffusion Transformers: Chimera uses sparse MoE to expand capacity while controlling activated compute in a hybrid KDA+MLA visual diffusion backbone (11B total / 2B activated). The paper fits the MoE size axis jointly with training tokens and image-vs-video data ratio under a three-axis Chinchilla law, giving one of the first filed compute-optimal allocations for a sparse-MoE video DiT rather than an LLM.
  • [2026-07-30] Introducing Inkling-Small — Thinking Machines' efficient open-weights MoE reasoning model: Inkling-Small (276B total / 12B active) inherits Inkling’s DeepSeek-V3-derived MoE routing wholesale — same 256 routed + 2 shared experts, same top-6 sigmoid + aux-free bias — with parameter count scaled down ~4×.
  • [2026-07-30] Local Mixtures of Experts: Essentially Free Test-Time Training via Model Merging: TTMM (Bertolissi, Hübotter, Hakimi, Krause) approaches MoE routing from the test-time-training direction: scale to ~100–1000 experts (two orders of magnitude beyond standard MoE) by embarrassingly parallel per-cluster LoRA training and replace conditional per-token routing with parameter merging of a small sparse subset selected once per prompt via sparse-softmax cross-attention over cluster centroids. Bypasses the load-balancing / router-capacity / token-level all-to-all issues that dominate the FFN-MoE thread — no LBL, no auxiliary loss, no router network, no per-token dispatch — at the cost of a coarser routing granularity (per prompt, not per token) and a training-data-clustering prior. Reports empirical saturation between 100 and 1000 experts on Wikipedia/GitHub-Python, offering the router-elimination camp (STEM: Scaling Transformers with Embedding Modules, Routing-Free Mixture-of-Experts) a fourth data point: even without a router, expert count is not free — data diversity, not routing capacity, becomes the ceiling.
  • [2026-07-27] Kimi K3 Technical Report — Open Frontier Intelligence: Kimi K3 tech report — deploys Quantile Balancing (bias derived from Top-(k+1) routing quantiles in a single forward pass) at 16-of-896 routing (sparsity 56), the deepest sparsity ratio in the cluster to date; also introduces Stable LatentMoE with RMSNorm between routed-expert aggregation and up-projection.
  • [2026-07-19] RynnBrain 1.1 — Open Embodied Foundation Models (Alibaba DAMO Academy): RynnBrain 1.1-122B-A10B is the first embodied-VLM at 122B-scale sparse MoE on this page: 122B total / ~10B activated, built on Qwen3.5-122B-A10B, released alongside dense 2B and 9B siblings under a shared training recipe to enable systematic scaling comparisons of embodied cognition, spatial reasoning, grounding, and planning.
  • [2026-07-16] Kimi K3 — Open Frontier Intelligence (2.8T MoE with KDA + AttnRes): Kimi K3 scales Quantile Balancing (Su, 2026-02) to 16-of-896 expert activation inside “Stable LatentMoE” at 2.8T parameters — the deepest sparsity ratio yet in the cluster to run in a shipping frontier model.
  • [2026-07-16] RxBrain: Embodied Cognition Foundation Model with Joint Language-Visual Reasoning and Imagination: RxBrain routes visual-understanding vs visual-generation tokens to separate FFN experts within a shared vision transformer (shared QKV), with the generation expert widened 2× (6,144→12,288 intermediate dim).
  • [2026-07-15] Introducing Inkling — Thinking Machines' open-weights 975B/41B MoE multimodal reasoning model: Inkling adopts the DeepSeek-V3 routing recipe verbatim at 975B-total / 41B-active scale: 256 routed experts + 2 shared, top-6 routed active per token, sigmoid router with auxiliary-loss-free load-balancing bias, and joint normalization of routed + shared expert scores before combining outputs. Concrete datapoint that the DeepSeek-V3 MoE recipe transfers to a from-scratch frontier training run by an independent lab.
  • [2026-07-14] Inside TPU and GPU Clusters: The Anatomy of Collective Communication: Grounds MoE Expert Parallelism in the underlying All-to-All primitive that dispatches tokens to expert-hosting devices: derives All-to-All as a “sharded transpose” on ring/path topologies (TPU) and shows that inside an NVSwitch node balanced All-to-All is ~2× faster than a bidirectional 1D torus ring at equal bandwidth. Complements DeepEP + NVSHMEM (which optimize the irregular per-token dispatch inside NCCL’s blind spot) by explaining what the balanced baseline looks like.
  • [2026-07-10] LingBot-VA 2.0: Native Video-Action Pretraining for Generalizable Robot Control: First filed instance of a DeepSeek-V3-style routed MoE (128 experts / top-8 / group-limited routing / auxfree Loss-Free-Balancing bias correction) inside a video-generation stream used as a policy backbone, rather than in an LLM or T2I model. Reports MoE-13B-A1.9B matching a Dense-5B baseline at both step-aligned training loss (dense slightly lower) and wall-clock-aligned final loss, and pairs it with a dense action expert under the Mixture-of-Transformers pattern — capacity added on the modeling-heavy modality only. Concrete datapoint that the LLM-side MoE recipe transfers cleanly to causal video-action pretraining without an optimization tax.
  • [2026-07-08] LingBot-Video: Scaling Mixture-of-Experts Video Pretraining for Embodied Intelligence: LingBot-Video is the first from-scratch MoE video-DiT datapoint in the cluster: imports DeepSeekMoE’s fine-grained-experts + shared-expert-isolation + auxiliary-loss-free bias + group-limited sigmoid routing directly into a video DiT (E=128, top-8), adds a sequence-wise Lseq to catch per-video routing imbalance that batch-level stats hide, and reports from-scratch scaling laws from 13B/A1.4B up to 120B/A11B — with fine-grained top-8-of-128 beating coarse top-4-of-64 under matched 13B total budget despite lower active FLOPs.
  • [2026-06-30] Agents-A1: A 35B MoE Agentic Foundation Model: Agents-A1 — another ~35B-total MoE entry in the open agentic-foundation cluster (alongside Qwen-AgentWorld-35B-A3B, Ornith-1.0-35B-MoE, Qwen3.6-35B-A3B), TP=1 serving config implies active-params fit a single GPU.
  • [2026-06-29] Demystifying NVSHMEM: A System-Level Analysis on Symmetric Memory and Device-Initiated Operations in GPU Communication: Uses DeepEP — the dispatch/combine library under every recent large MoE training/serving stack — as a case study for NVSHMEM’s device-side symmetric-memory model; explains why standard NCCL all-to-all is the wrong primitive for irregular per-token MoE dispatch and what NVSHMEM provides instead.
  • [2026-06-23] PithTrain: A Compact and Agent-Native MoE Training System: PithTrain provides a compact (11K-LoC) production-quality MoE training framework targeting Qwen3-MoE, GPT-OSS, and DeepSeek-V2-Lite on H100/B200 with EP-dispatch deduplication, FP8 weight cache across micro-batches, and DualPipeV-based EP all-to-all overlap on a separate communication stream. Matches Megatron-LM throughput on 4 of 5 configurations while shipping pure-Python self-contained per-model files under models/ (no shared layer skeleton or plugin registry) — a deliberate trade of cross-model reuse for local readability and agent-traceability.
  • [2026-05-28] Paris 2.0 — decentralized trained video generation model from Bagel Labs (bidhan announcement): Paris 2.0 announcement extends the router-over-independently-trained-experts design (no load balancing, no joint training) from image diffusion to video generation, claiming ~2× FVD over a monolithic iso-compute baseline.
  • [2026-05-24] ERNIE 4.5 Technical Report: ERNIE 4.5 Technical Report (Baidu, June 2025). Heterogeneous fine-grained MoE with text experts, vision experts (1/3 the intermediate dim of text experts, removed in the final layer), and a tier of shared experts; routing is modality-isolated — each token only sees experts of its own modality, plus the always-on shared experts; self-attention parameters are shared. Author claim is that text and vision expert activation patterns are intrinsically different (heatmap evidence: concentrated for text, dispersed for vision) so a single router collapses under multimodal continuation training. Adds two loss-side ingredients: a router-orthogonality auxiliary loss and a token-balanced multimodal LBL that compensates for the text-vs-visual token-count imbalance. Engineering: intra-node Expert Parallelism, FP8 mixed precision, hits 47% MFU on 2,016 H800s pretraining the largest (300B-A47B) language model. Three new axes for this page: modality-conditioned routing, router-direction orthogonalization, token-count-aware LBL.
  • [2026-05-24] MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production: ByteDance Seed’s production MoE training system. Pairs Ulysses-style Sequence Parallelism for attention with Expert Parallelism for FFN, drops Tensor Parallelism entirely, and rejects Context Parallelism on causal-mask-imbalance grounds. Reports 1.88× MFU over Megatron-LM on a 352B MoE / 1,440 Hopper GPUs. Algorithmic-systems counterpart to Scalable Training of Mixture-of-Experts Models with Megatron Core‘s Parallel Folding (same TP-vs-MoE mismatch problem, opposite resolution) and sibling to veScale-FSDP: Flexible and High-Performance FSDP at Scale (same ByteDance team, different layer of the stack). Per-GPU-group auxiliary balance loss adopts DeepSeek-V2’s recipe and sits between the micro-batch and global-batch LBL scopes in the Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models axis.
  • [2026-05-24] Slicing and Dicing MoEs — Margaret Li announcing a >2000-MoE-LM design-space study: Margaret Li announces “Slicing and Dicing MoEs”, a study that reports training >2000 MoE LMs across five design axes (total vs active experts, expert size, shared experts, routing, token dropping) and summarizes the sweep with a one-line ranking: expert size and count dominate. First artifact in the cluster to claim a multi-axis head-to-head; arxiv link not yet visible. Candidate empirical answer to the “multi-axis Pareto frontier is unmapped” open question below.
  • [2026-05-24] Raven Part 1 — Memory as a Set of Slots: Raven exports the MoE-routing vocabulary from FFN-experts to state-slots. Unifies SSMs (dense router rt=1r_t = \mathbf{1}) and SWA (one-hot cyclic router rt=etr_t = e_t) under a single state-update equation; Raven uses a learned sparse rtr_t to decide per-token which slots get the SSM decay-and-write and which are preserved verbatim. Imports the same questions the FFN-MoE thread is wrestling with (router sparsity, load balancing, top-k vs soft) into recurrent-state design.
  • [2026-05-24] Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models: shows that the canonical LBL implementation is computed at micro-batch (≈ sequence-level) scope and silently forbids domain-specialized routing; a one-line all-reduce on fif_i to make balance corpus-level beats both micro-batch LBL and DeepSeek’s loss-free bias-update on both PPL and downstream accuracy up to 42.8B params, and produces visibly domain-specialized expert heatmaps.
  • [2026-05-24] Routing-Free Mixture-of-Experts: composes AoE (expert-internal scoring) with ReMoE (ReLU gating, no TopK/Softmax) and adds a per-expert bias to give a fully decentralized self-activation rule; pairs this with a unified β\beta-interpolated token+expert balancing loss and adaptive auxiliary coefficient; reports scaling gains at 0.8B and wider LR tolerance.
  • [2026-05-23] Path-Constrained Mixture-of-Experts: PathMoE shares router parameters across blocks of consecutive layers, sitting between independent-per-layer (standard MoE) and one-shared-router (Omni-Router ASR) on the constraint spectrum; analytically demonstrates that this amplifies natural path concentration.
  • [2026-05-23] Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design: fits a closed-form scaling law for r∗r^* = fraction of per-token FLOPs spent in experts as a function of CC and sparsity SS; r∗r^* rises with compute.
  • [2026-05-23] Scalable Training of Mixture-of-Experts Models with Megatron Core: Megatron-Core MoE systems report — names the attention-vs-experts compute mismatch as the systems wall and provides Parallel Folding to resolve it.
  • [2026-05-23] DynaMoE: Dynamic Token-Level Expert Activation with Layer-Wise Adaptive Capacity for Mixture-of-Experts Neural Networks: percentile-threshold gate replaces TopK and sweeps six closed-form per-layer expert-count schedules at toy scale (<6M params); finds optimal schedule is task- and scale-dependent.
  • [2026-05-23] Quantile Balancing: A Hyperparameter-Free MoE Load Balancing Method: removes the auxiliary-loss / bias-update tuning axis via quantile balancing at 32B-A5B production scale.
  • [2026-05-22] STEM: Scaling Transformers with Embedding Modules: replaces the FFN with a static per-token embedding lookup, eliminating routing entirely — adjacent / opposite-direction alternative to the router-redesign thread.

Open questions

Section titled “Open questions”
  • Does eliminating routing transfer past 1B? Routing-Free MoE’s most aggressive design (no router, no TopK, no Softmax, no per-layer sparsity) reports gains widening at 0.8B but no run exists at frontier scale (>10B). PathMoE, the Su quantile recipe, and Demons-in-the-Detail’s global-batch LBL each have at least one 30B+-class datapoint; Routing-Free MoE does not.
  • Pretrained-MoE conversion. Every paper in the cluster trains from scratch. Adapting a pretrained Mixtral/DeepSeek MoE into router-free, path-constrained, global-batch-balanced, or quantile-balanced form via continued pretraining is explicitly future work everywhere — and the cost calculus changes radically if such conversion is cheap.
  • Composition. PathMoE constrains the router parameters, Routing-Free MoE removes the router, Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design prescribes r∗r^*, Quantile Balancing removes the LB knob, and Demons-in-the-Detail rescopes the LB. The papers do not run head-to-head against each other or in combination; the multi-axis Pareto frontier is unmapped. Slicing and Dicing MoEs — Margaret Li announcing a >2000-MoE-LM design-space study claims a >2000-run multi-axis sweep with size + count as the dominant levers — once the underlying paper surfaces, it may close (or refine) this question.
  • Per-layer vs. global density: Routing-Free MoE’s 11-point PPL swing from global density enforcement is the cluster’s biggest single result, but only one paper measured it. Does the per-layer-uniform-sparsity bias hurt standard MoE / AoE / ReMoE by similar amounts? Demons-in-the-Detail’s temporal analogue — global vs. per-step balance scope — suggests the same answer holds along the time axis.
  • Path-distribution concentration as a diagnostic. PathMoE introduces path-occupancy and path-stability as metrics for routing quality; Demons-in-the-Detail’s expert-frequency-per-domain heatmaps are a related diagnostic. Could a path-concentration or domain-specialization metric replace the validation-perplexity proxy as the primary signal for routing design?
  • MoE-over-state vs. MoE-over-FFN. Raven applies the same routing primitives to the recurrent state. Open question: do the lessons from the FFN-side thread (global-batch LBL, soft-gate-not-TopK, per-layer-uniform-sparsity hurts, r∗r^* scaling law) transfer to slot routing in SSMs/Raven, or does the temporal structure of state updates change the answer?
  • Drop-TP vs Parallel-Folding for the attention/MoE parallelism mismatch. MegaScale-MoE (MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production) and Megatron-Core MoE (Scalable Training of Mixture-of-Experts Models with Megatron Core) resolve the same problem with opposite recipes — drop TP for attention vs. keep TP/CP for attention but decouple parallelism groups. Both ship to production. No head-to-head comparison exists; the right answer likely depends on context length (CP / ring attention is argued necessary at ultra-long contexts, but rejected by MegaScale-MoE on causal-mask-imbalance grounds at production-typical lengths) and on intra-node topology (NVL8 vs NVL72).
  • Modality-conditioned routing vs single-router multimodal MoE. ERNIE 4.5 (ERNIE 4.5 Technical Report) argues for separate text-expert and vision-expert pools on the basis that text and vision activation patterns differ (concentrated vs dispersed) and a single router collapses. Open: at what model scale does single-router multimodal MoE collapse, vs when is the routing capacity sufficient to handle both modalities? No filed head-to-head comparison exists. Adjacent open question: is modality isolation a capacity fix (more routers per modality) or a gradient-interference fix (separate expert pools so that modality-specific FFN weights don’t get overwritten)?

Papers

Section titled “Papers”

2026-11

Section titled “2026-11”

2026-05

Section titled “2026-05”

2026-04

Section titled “2026-04”

2026-03

Section titled “2026-03”

2026-02

Section titled “2026-02”

2026-01

Section titled “2026-01”

2025-12

Section titled “2025-12”

2025-11

Section titled “2025-11”

2025-10

Section titled “2025-10”

2025-09

Section titled “2025-09”

2025-08

Section titled “2025-08”

2025-07

Section titled “2025-07”

2025-06

Section titled “2025-06”

2025-05

Section titled “2025-05”

2025-01

Section titled “2025-01”

2024-01

Section titled “2024-01”

Unknown date

Section titled “Unknown date”