MoE Routing Design

TL;DR

A cluster of 2026 papers questioning the rigid inductive biases of the standard Mixture-of-Experts routing pipeline — external router, TopK, Softmax, hard-coded per-layer expert counts, fixed attention/FFN FLOPs split, auxiliary load-balancing loss with a tuned coefficient. Each contribution removes or replaces one (or more) of these design axes: the architecture-side end ranges from share router weights across layers (PathMoE) to eliminate the router entirely (Routing-Free MoE, STEM); the training-side end ranges from unify token-balancing and expert-balancing to make the load-balancing coefficient adaptive to give an explicit scaling law for attention vs. expert FLOPs allocation. Common thread: the empirical token-path distribution is highly non-uniform, and constraints that fight that non-uniformity (uniform per-layer sparsity, fixed TopK, equal-FLOPs-per-token) leave loss on the table.

Key claims

• Standard MoE has at least four un-motivated rigid biases: TopK selection forces fixed sparsity regardless of input complexity, Softmax discards absolute magnitude information via competitive normalization, the external router has tiny capacity relative to the experts it scores, and per-layer expert counts are usually held uniform (Routing-Free Mixture-of-Experts §1; DynaMoE: Dynamic Token-Level Expert Activation with Layer-Wise Adaptive Capacity for Mixture-of-Experts Neural Networks §1; Path-Constrained Mixture-of-Experts §1).
• The load-balancing loss has a fifth, more silent bias: the scope over which $f_i$ is computed. Standard frameworks compute LBL inside a micro-batch (only a few sequences at billion-scale), making the LBL effectively a per-sequence uniformity constraint that forbids any sequence — including a clearly domain-specific one — from preferring a subset of experts (Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models §1, Fig. 1).
• Replacing TopK with a soft gate (ReLU, percentile threshold) recovers per-token-adaptive sparsity and preserves activation magnitudes (Routing-Free Mixture-of-Experts §3.1, builds on ReMoE; DynaMoE: Dynamic Token-Level Expert Activation with Layer-Wise Adaptive Capacity for Mixture-of-Experts Neural Networks §3.2 via percentile gate).
• Expert-internal scoring (each expert scores itself from its own low-rank projection) eliminates the router-capacity bottleneck while preserving per-expert specialization signal (Routing-Free Mixture-of-Experts §3.1, building on AoE; Lv et al. 2025).
• Sharing router parameters across contiguous layer blocks amplifies the natural path-distribution concentration rather than fighting it — yields better cross-layer routing consistency and robustness to input perturbation (Path-Constrained Mixture-of-Experts Abstract, §1).
• Per-layer uniform sparsity is empirically harmful: enforcing the activation-density target globally rather than per-layer drops perplexity by ~11 points at scale S, and the trained model self-organizes into a non-uniform depth-wise activation pattern (Routing-Free Mixture-of-Experts §5.1, Appendix D.2; consistent with the Information-Bottleneck rationale in DynaMoE: Dynamic Token-Level Expert Activation with Layer-Wise Adaptive Capacity for Mixture-of-Experts Neural Networks §7.1.1).
• Token-balancing and expert-balancing are complementary, not mutually exclusive: a single $\beta$-interpolated load-balancing loss with $\beta=0.5$ beats both pure extremes on perplexity and throughput (Routing-Free Mixture-of-Experts §5.2, Table 4; consistent with Muennighoff et al. 2025 cited there).
• The load-balancing coefficient can be made adaptive rather than a static hyperparameter — driven by feedback from empirical activation density vs. target — removing one tuning knob (Routing-Free Mixture-of-Experts §3.2 Eq. 17, building on ReMoE; Quantile Balancing: A Hyperparameter-Free MoE Load Balancing Method removes a related knob via quantile balancing without auxiliary loss).
• Global-batch LBL beats both micro-batch LBL and DeepSeek-style auxiliary-loss-free bias updates on both pre-training PPL and downstream task accuracy, up to 42.8B parameters — and the improvement comes with visibly more domain-specialized expert routing (Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models §4.2, §4.3, Fig. 1).
• The optimal attention-vs-experts FLOPs allocation $r^*$ is not a constant but follows a power law in total compute and sparsity: as $C$ grows, capacity should shift toward experts (Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design §3, Fig. 5).
• Systems-level wall for MoE-at-scale is the attention-vs-experts compute mismatch (attention is TP-friendly, MoE is EP-friendly); algorithmic prescriptions for $r^*$ (Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design) compose with systems recipes like Parallel Folding (Scalable Training of Mixture-of-Experts Models with Megatron Core §3) and with the orthogonal MegaScale-MoE recipe of dropping TP entirely in favor of Ulysses-style SP for attention plus EP for FFN (MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production §3).
• At production scale on Hopper, intra-node TP for MoE attention becomes a net loss — its all-gather/reduce-scatter on the critical path can exceed self-attention compute time, motivating the SP+EP+PP combination at the cost of small extra parameter-synchronization overhead (MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production §3.1).
• An extreme alternative is to eliminate the router entirely by replacing the FFN with a static per-token embedding lookup — sidesteps load balancing, all-to-all communication, and routing decisions but requires retraining (STEM: Scaling Transformers with Embedding Modules).
• At scale, removing routing inductive biases improves training-stability margins: Routing-Free MoE tolerates a wider LR range before collapse and continues improving with scale where standard MoE plateaus (Routing-Free Mixture-of-Experts §4.4, Fig. 5).
• The routing-as-design vocabulary extends past the FFN: Raven applies the same primitives — sparse data-dependent router, expert/slot specialization, write/persist decision per “expert” — to the recurrent hidden state, casting memory allocation in linear-time sequence models as an MoE-over-slots routing problem (Raven Part 1 — Memory as a Set of Slots).
• Author claim from a large-scale sweep: expert size and count dominate the other MoE design axes. A >2000-MoE-LM study (“Slicing and Dicing MoEs”, Margaret Li et al.) summarizes its multi-axis sweep — total vs active experts, expert size, shared experts, routing, token dropping — with the takeaway that size and count are the primary levers (Slicing and Dicing MoEs — Margaret Li announcing a >2000-MoE-LM design-space study). Filed from the announcement tweet; underlying paper not yet linked.
• Per-GPU-group balancing as the production default: rather than balancing the per-individual-expert frequency, MegaScale-MoE treats co-located experts on each device as a single virtual expert for the auxiliary-loss computation, following DeepSeek-V2 (MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production §3.2). Sits in the design space between micro-batch-scoped LBL and the global-batch-scoped LBL that Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models advocates.
• Modality-isolated routing as another lever: when a single MoE has to serve multiple modalities, partitioning the experts by modality (separate text and vision expert pools, plus a shared-expert tier, sharing only self-attention parameters) avoids the routing-collapse failure mode that extending a text-only MoE into multimodal training otherwise produces (ERNIE 4.5 Technical Report §2.1, Fig. 2). ERNIE 4.5 pairs this with a router-orthogonalization auxiliary loss (decorrelating expert-selection directions) and a token-balanced multimodal LBL (compensating for token-count imbalance between text and visual streams) (ERNIE 4.5 Technical Report §3.4.1, §3.4.2). Adds three new axes — modality-conditioned routing, router-direction orthogonalization, and token-count-aware LBL — to the design-space vocabulary, complementary to PathMoE’s layer-shared routing and Demons-in-the-Detail’s global-batch LBL rescoping.

Recent contributions

• [2026-07-19] RynnBrain 1.1 — Open Embodied Foundation Models (Alibaba DAMO Academy): RynnBrain 1.1-122B-A10B is the first embodied-VLM at 122B-scale sparse MoE on this page: 122B total / ~10B activated, built on Qwen3.5-122B-A10B, released alongside dense 2B and 9B siblings under a shared training recipe to enable systematic scaling comparisons of embodied cognition, spatial reasoning, grounding, and planning.
• [2026-07-16] Kimi K3 — Open Frontier Intelligence (2.8T MoE with KDA + AttnRes): Kimi K3 scales Quantile Balancing (Su, 2026-02) to 16-of-896 expert activation inside “Stable LatentMoE” at 2.8T parameters — the deepest sparsity ratio yet in the cluster to run in a shipping frontier model.
• [2026-07-16] RxBrain: Embodied Cognition Foundation Model with Joint Language-Visual Reasoning and Imagination: RxBrain routes visual-understanding vs visual-generation tokens to separate FFN experts within a shared vision transformer (shared QKV), with the generation expert widened 2× (6,144→12,288 intermediate dim).
• [2026-07-15] Introducing Inkling — Thinking Machines' open-weights 975B/41B MoE multimodal reasoning model: Inkling adopts the DeepSeek-V3 routing recipe verbatim at 975B-total / 41B-active scale: 256 routed experts + 2 shared, top-6 routed active per token, sigmoid router with auxiliary-loss-free load-balancing bias, and joint normalization of routed + shared expert scores before combining outputs. Concrete datapoint that the DeepSeek-V3 MoE recipe transfers to a from-scratch frontier training run by an independent lab.
• [2026-07-14] Inside TPU and GPU Clusters: The Anatomy of Collective Communication: Grounds MoE Expert Parallelism in the underlying All-to-All primitive that dispatches tokens to expert-hosting devices: derives All-to-All as a “sharded transpose” on ring/path topologies (TPU) and shows that inside an NVSwitch node balanced All-to-All is ~2× faster than a bidirectional 1D torus ring at equal bandwidth. Complements DeepEP + NVSHMEM (which optimize the irregular per-token dispatch inside NCCL’s blind spot) by explaining what the balanced baseline looks like.
• [2026-07-10] LingBot-VA 2.0: Native Video-Action Pretraining for Generalizable Robot Control: First filed instance of a DeepSeek-V3-style routed MoE (128 experts / top-8 / group-limited routing / auxfree Loss-Free-Balancing bias correction) inside a video-generation stream used as a policy backbone, rather than in an LLM or T2I model. Reports MoE-13B-A1.9B matching a Dense-5B baseline at both step-aligned training loss (dense slightly lower) and wall-clock-aligned final loss, and pairs it with a dense action expert under the Mixture-of-Transformers pattern — capacity added on the modeling-heavy modality only. Concrete datapoint that the LLM-side MoE recipe transfers cleanly to causal video-action pretraining without an optimization tax.
• [2026-07-08] LingBot-Video: Scaling Mixture-of-Experts Video Pretraining for Embodied Intelligence: LingBot-Video is the first from-scratch MoE video-DiT datapoint in the cluster: imports DeepSeekMoE’s fine-grained-experts + shared-expert-isolation + auxiliary-loss-free bias + group-limited sigmoid routing directly into a video DiT (E=128, top-8), adds a sequence-wise Lseq to catch per-video routing imbalance that batch-level stats hide, and reports from-scratch scaling laws from 13B/A1.4B up to 120B/A11B — with fine-grained top-8-of-128 beating coarse top-4-of-64 under matched 13B total budget despite lower active FLOPs.
• [2026-06-30] Agents-A1: A 35B MoE Agentic Foundation Model: Agents-A1 — another ~35B-total MoE entry in the open agentic-foundation cluster (alongside Qwen-AgentWorld-35B-A3B, Ornith-1.0-35B-MoE, Qwen3.6-35B-A3B), TP=1 serving config implies active-params fit a single GPU.
• [2026-06-29] Demystifying NVSHMEM: A System-Level Analysis on Symmetric Memory and Device-Initiated Operations in GPU Communication: Uses DeepEP — the dispatch/combine library under every recent large MoE training/serving stack — as a case study for NVSHMEM’s device-side symmetric-memory model; explains why standard NCCL all-to-all is the wrong primitive for irregular per-token MoE dispatch and what NVSHMEM provides instead.
• [2026-06-23] PithTrain: A Compact and Agent-Native MoE Training System: PithTrain provides a compact (11K-LoC) production-quality MoE training framework targeting Qwen3-MoE, GPT-OSS, and DeepSeek-V2-Lite on H100/B200 with EP-dispatch deduplication, FP8 weight cache across micro-batches, and DualPipeV-based EP all-to-all overlap on a separate communication stream. Matches Megatron-LM throughput on 4 of 5 configurations while shipping pure-Python self-contained per-model files under models/ (no shared layer skeleton or plugin registry) — a deliberate trade of cross-model reuse for local readability and agent-traceability.
• [2026-05-28] Paris 2.0 — decentralized trained video generation model from Bagel Labs (bidhan announcement): Paris 2.0 announcement extends the router-over-independently-trained-experts design (no load balancing, no joint training) from image diffusion to video generation, claiming ~2× FVD over a monolithic iso-compute baseline.
• [2026-05-24] ERNIE 4.5 Technical Report: ERNIE 4.5 Technical Report (Baidu, June 2025). Heterogeneous fine-grained MoE with text experts, vision experts (1/3 the intermediate dim of text experts, removed in the final layer), and a tier of shared experts; routing is modality-isolated — each token only sees experts of its own modality, plus the always-on shared experts; self-attention parameters are shared. Author claim is that text and vision expert activation patterns are intrinsically different (heatmap evidence: concentrated for text, dispersed for vision) so a single router collapses under multimodal continuation training. Adds two loss-side ingredients: a router-orthogonality auxiliary loss and a token-balanced multimodal LBL that compensates for the text-vs-visual token-count imbalance. Engineering: intra-node Expert Parallelism, FP8 mixed precision, hits 47% MFU on 2,016 H800s pretraining the largest (300B-A47B) language model. Three new axes for this page: modality-conditioned routing, router-direction orthogonalization, token-count-aware LBL.
• [2026-05-24] MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production: ByteDance Seed’s production MoE training system. Pairs Ulysses-style Sequence Parallelism for attention with Expert Parallelism for FFN, drops Tensor Parallelism entirely, and rejects Context Parallelism on causal-mask-imbalance grounds. Reports 1.88× MFU over Megatron-LM on a 352B MoE / 1,440 Hopper GPUs. Algorithmic-systems counterpart to Scalable Training of Mixture-of-Experts Models with Megatron Core‘s Parallel Folding (same TP-vs-MoE mismatch problem, opposite resolution) and sibling to veScale-FSDP: Flexible and High-Performance FSDP at Scale (same ByteDance team, different layer of the stack). Per-GPU-group auxiliary balance loss adopts DeepSeek-V2’s recipe and sits between the micro-batch and global-batch LBL scopes in the Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models axis.
• [2026-05-24] Slicing and Dicing MoEs — Margaret Li announcing a >2000-MoE-LM design-space study: Margaret Li announces “Slicing and Dicing MoEs”, a study that reports training >2000 MoE LMs across five design axes (total vs active experts, expert size, shared experts, routing, token dropping) and summarizes the sweep with a one-line ranking: expert size and count dominate. First artifact in the cluster to claim a multi-axis head-to-head; arxiv link not yet visible. Candidate empirical answer to the “multi-axis Pareto frontier is unmapped” open question below.
• [2026-05-24] Raven Part 1 — Memory as a Set of Slots: Raven exports the MoE-routing vocabulary from FFN-experts to state-slots. Unifies SSMs (dense router $r_t = \mathbf{1}$) and SWA (one-hot cyclic router $r_t = e_t$) under a single state-update equation; Raven uses a learned sparse $r_t$ to decide per-token which slots get the SSM decay-and-write and which are preserved verbatim. Imports the same questions the FFN-MoE thread is wrestling with (router sparsity, load balancing, top-k vs soft) into recurrent-state design.
• [2026-05-24] Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models: shows that the canonical LBL implementation is computed at micro-batch (≈ sequence-level) scope and silently forbids domain-specialized routing; a one-line all-reduce on $f_i$ to make balance corpus-level beats both micro-batch LBL and DeepSeek’s loss-free bias-update on both PPL and downstream accuracy up to 42.8B params, and produces visibly domain-specialized expert heatmaps.
• [2026-05-24] Routing-Free Mixture-of-Experts: composes AoE (expert-internal scoring) with ReMoE (ReLU gating, no TopK/Softmax) and adds a per-expert bias to give a fully decentralized self-activation rule; pairs this with a unified $\beta$-interpolated token+expert balancing loss and adaptive auxiliary coefficient; reports scaling gains at 0.8B and wider LR tolerance.
• [2026-05-23] Path-Constrained Mixture-of-Experts: PathMoE shares router parameters across blocks of consecutive layers, sitting between independent-per-layer (standard MoE) and one-shared-router (Omni-Router ASR) on the constraint spectrum; analytically demonstrates that this amplifies natural path concentration.
• [2026-05-23] Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design: fits a closed-form scaling law for $r^*$ = fraction of per-token FLOPs spent in experts as a function of $C$ and sparsity $S$; $r^*$ rises with compute.
• [2026-05-23] Scalable Training of Mixture-of-Experts Models with Megatron Core: Megatron-Core MoE systems report — names the attention-vs-experts compute mismatch as the systems wall and provides Parallel Folding to resolve it.
• [2026-05-23] DynaMoE: Dynamic Token-Level Expert Activation with Layer-Wise Adaptive Capacity for Mixture-of-Experts Neural Networks: percentile-threshold gate replaces TopK and sweeps six closed-form per-layer expert-count schedules at toy scale (<6M params); finds optimal schedule is task- and scale-dependent.
• [2026-05-23] Quantile Balancing: A Hyperparameter-Free MoE Load Balancing Method: removes the auxiliary-loss / bias-update tuning axis via quantile balancing at 32B-A5B production scale.
• [2026-05-22] STEM: Scaling Transformers with Embedding Modules: replaces the FFN with a static per-token embedding lookup, eliminating routing entirely — adjacent / opposite-direction alternative to the router-redesign thread.

Open questions

• Does eliminating routing transfer past 1B? Routing-Free MoE’s most aggressive design (no router, no TopK, no Softmax, no per-layer sparsity) reports gains widening at 0.8B but no run exists at frontier scale (>10B). PathMoE, the Su quantile recipe, and Demons-in-the-Detail’s global-batch LBL each have at least one 30B+-class datapoint; Routing-Free MoE does not.
• Pretrained-MoE conversion. Every paper in the cluster trains from scratch. Adapting a pretrained Mixtral/DeepSeek MoE into router-free, path-constrained, global-batch-balanced, or quantile-balanced form via continued pretraining is explicitly future work everywhere — and the cost calculus changes radically if such conversion is cheap.
• Composition. PathMoE constrains the router parameters, Routing-Free MoE removes the router, Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design prescribes $r^*$, Quantile Balancing removes the LB knob, and Demons-in-the-Detail rescopes the LB. The papers do not run head-to-head against each other or in combination; the multi-axis Pareto frontier is unmapped. Slicing and Dicing MoEs — Margaret Li announcing a >2000-MoE-LM design-space study claims a >2000-run multi-axis sweep with size + count as the dominant levers — once the underlying paper surfaces, it may close (or refine) this question.
• Per-layer vs. global density: Routing-Free MoE’s 11-point PPL swing from global density enforcement is the cluster’s biggest single result, but only one paper measured it. Does the per-layer-uniform-sparsity bias hurt standard MoE / AoE / ReMoE by similar amounts? Demons-in-the-Detail’s temporal analogue — global vs. per-step balance scope — suggests the same answer holds along the time axis.
• Path-distribution concentration as a diagnostic. PathMoE introduces path-occupancy and path-stability as metrics for routing quality; Demons-in-the-Detail’s expert-frequency-per-domain heatmaps are a related diagnostic. Could a path-concentration or domain-specialization metric replace the validation-perplexity proxy as the primary signal for routing design?
• MoE-over-state vs. MoE-over-FFN. Raven applies the same routing primitives to the recurrent state. Open question: do the lessons from the FFN-side thread (global-batch LBL, soft-gate-not-TopK, per-layer-uniform-sparsity hurts, $r^*$ scaling law) transfer to slot routing in SSMs/Raven, or does the temporal structure of state updates change the answer?
• Drop-TP vs Parallel-Folding for the attention/MoE parallelism mismatch. MegaScale-MoE (MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production) and Megatron-Core MoE (Scalable Training of Mixture-of-Experts Models with Megatron Core) resolve the same problem with opposite recipes — drop TP for attention vs. keep TP/CP for attention but decouple parallelism groups. Both ship to production. No head-to-head comparison exists; the right answer likely depends on context length (CP / ring attention is argued necessary at ultra-long contexts, but rejected by MegaScale-MoE on causal-mask-imbalance grounds at production-typical lengths) and on intra-node topology (NVL8 vs NVL72).
• Modality-conditioned routing vs single-router multimodal MoE. ERNIE 4.5 (ERNIE 4.5 Technical Report) argues for separate text-expert and vision-expert pools on the basis that text and vision activation patterns differ (concentrated vs dispersed) and a single router collapses. Open: at what model scale does single-router multimodal MoE collapse, vs when is the routing capacity sufficient to handle both modalities? No filed head-to-head comparison exists. Adjacent open question: is modality isolation a capacity fix (more routers per modality) or a gradient-interference fix (separate expert pools so that modality-specific FFN weights don’t get overwritten)?

Papers

2026-11

• Kimi K2 Thinking — 1T MoE thinking model with native INT4 QAT and 200–300-step tool use — Kimi K2 Thinking — 1T MoE thinking model with native INT4 QAT and 200–300-step tool use (published 2026-11-06)

2026-05

• Introducing Composer 2.5 — Introducing Composer 2.5 (published 2026-05-18)
• Slicing and Dicing MoEs — Margaret Li announcing a >2000-MoE-LM design-space study — Slicing and Dicing MoEs — Margaret Li announcing a >2000-MoE-LM design-space study (published 2026-05-18)
• Interaction Models: A Scalable Approach to Human-AI Collaboration — Interaction Models: A Scalable Approach to Human-AI Collaboration (published 2026-05-11)
• SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training — SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training (published 2026-05-09)
• ZAYA1-8B Technical Report — ZAYA1-8B Technical Report (published 2026-05-06)
• How to Make LLM Training Faster with Unsloth and NVIDIA — How to Make LLM Training Faster with Unsloth and NVIDIA (published 2026-05-06)

2026-04

• DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models — DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models (published 2026-04-15)
• Raven Part 1 — Memory as a Set of Slots — Raven Part 1 — Memory as a Set of Slots (published 2026-04-10)
• Routing-Free Mixture-of-Experts — Routing-Free Mixture-of-Experts (published 2026-04-01)

2026-03

• Path-Constrained Mixture-of-Experts — Path-Constrained Mixture-of-Experts (published 2026-03-18)
• Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design — Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design (published 2026-03-11)
• Scalable Training of Mixture-of-Experts Models with Megatron Core — Scalable Training of Mixture-of-Experts Models with Megatron Core (published 2026-03-08)
• DynaMoE: Dynamic Token-Level Expert Activation with Layer-Wise Adaptive Capacity for Mixture-of-Experts Neural Networks — DynaMoE: Dynamic Token-Level Expert Activation with Layer-Wise Adaptive Capacity for Mixture-of-Experts Neural Networks (published 2026-03-02)
• CausalWan2.2-I2V-A14B-Preview — FastVideo causal I2V model on Wan2.2-A14B — CausalWan2.2-I2V-A14B-Preview — FastVideo causal I2V model on Wan2.2-A14B (published 2026-03-02)

2026-02

• veScale-FSDP: Flexible and High-Performance FSDP at Scale — veScale-FSDP: Flexible and High-Performance FSDP at Scale (published 2026-02-25)
• Quantile Balancing: A Hyperparameter-Free MoE Load Balancing Method — Quantile Balancing: A Hyperparameter-Free MoE Load Balancing Method (published 2026-02-01)

2026-01

• STEM: Scaling Transformers with Embedding Modules — STEM: Scaling Transformers with Embedding Modules (published 2026-01-15)

2025-12

• SonicMoE: Accelerating MoE with IO and Tile-aware Optimizations — SonicMoE: Accelerating MoE with IO and Tile-aware Optimizations (published 2025-12-16)
• Fuli Luo on MiMo-V2-Flash engineering — Hybrid SWA, MTP, MOPD post-training — Fuli Luo on MiMo-V2-Flash engineering — Hybrid SWA, MTP, MOPD post-training (published 2025-12-16)
• NVIDIA Debuts Nemotron 3 Family of Open Models — NVIDIA Debuts Nemotron 3 Family of Open Models (published 2025-12-15)
• EMMA: Efficient Multimodal Understanding, Generation, and Editing with a Unified Architecture — EMMA: Efficient Multimodal Understanding, Generation, and Editing with a Unified Architecture (published 2025-12-04)
• Mistral 3 release — vision-capable 3B/8B/14B dense + Large 3 (675B total, 41B active MoE) under Apache 2.0 — Mistral 3 release — vision-capable 3B/8B/14B dense + Large 3 (675B total, 41B active MoE) under Apache 2.0 (published 2025-12-02)
• The Trinity Manifesto — Arcee's open-weight MoE family (Mini + Nano + Large roadmap) — The Trinity Manifesto — Arcee’s open-weight MoE family (Mini + Nano + Large roadmap) (published 2025-12-01)

2025-11

• LFM2 Technical Report — LFM2 Technical Report (published 2025-11-28)
• Qwen3-VL Technical Report — Qwen3-VL Technical Report (published 2025-11-26)
• Mixture of States: Routing Token-Level Dynamics for Multimodal Generation — Mixture of States: Routing Token-Level Dynamics for Multimodal Generation (published 2025-11-20)
• The MoE 101 Guide: From Theory to Production (Daria Soboleva, Cerebras) — The MoE 101 Guide: From Theory to Production (Daria Soboleva, Cerebras) (published 2025-11-14)

2025-10

• Phased DMD: Few-step Distribution Matching Distillation via Score Matching within Subintervals — Phased DMD: Few-step Distribution Matching Distillation via Score Matching within Subintervals (published 2025-10-31)
• LongCat-Flash-Omni Technical Report — LongCat-Flash-Omni Technical Report (published 2025-10-31)
• Kimi Linear: An Expressive, Efficient Attention Architecture — Kimi Linear: An Expressive, Efficient Attention Architecture (published 2025-10-30)
• Ming-Flash-Omni: A Sparse, Unified Architecture for Multimodal Perception and Generation — Ming-Flash-Omni: A Sparse, Unified Architecture for Multimodal Perception and Generation (published 2025-10-28)
• Paris: A Decentralized Trained Open-Weight Diffusion Model — Paris: A Decentralized Trained Open-Weight Diffusion Model (published 2025-10-03)
• lmms-engine — A simple, unified multimodal models training engine — lmms-engine — A simple, unified multimodal models training engine (published 2025-10-01)

2025-09

• Qwen3-VL — Multimodal LLM series (Dense + MoE, Instruct + Thinking) — Qwen3-VL — Multimodal LLM series (Dense + MoE, Instruct + Thinking) (published 2025-09-23)
• SAIL-VL2 Technical Report — SAIL-VL2 Technical Report (published 2025-09-18)
• Qwen3-Next-80B-A3B — hybrid Gated DeltaNet + Gated Attention, ultra-sparse 512-expert MoE, MTP (Alibaba Qwen announcement) — Qwen3-Next-80B-A3B — hybrid Gated DeltaNet + Gated Attention, ultra-sparse 512-expert MoE, MTP (Alibaba Qwen announcement) (published 2025-09-11)
• LongCat-Flash Technical Report — LongCat-Flash Technical Report (published 2025-09-01)

2025-08

• DeepSeek-V3.1-Base — long-context-extended V3 base with UE8M0 FP8 microscaling — DeepSeek-V3.1-Base — long-context-extended V3 base with UE8M0 FP8 microscaling (published 2025-08-21)
• DeepSeek-V3.1 — hybrid thinking + non-thinking model, smarter tool calling, faster thinking — DeepSeek-V3.1 — hybrid thinking + non-thinking model, smarter tool calling, faster thinking (published 2025-08-21)
• GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models — GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models (published 2025-08-08)
• GLM-4.5: Agentic Foundation Model from Z.ai — GLM-4.5: Agentic Foundation Model from Z.ai (published 2025-08-08)
• VeOmni: Scaling Any Modality Model Training with Model-Centric Distributed Recipe Zoo — VeOmni: Scaling Any Modality Model Training with Model-Centric Distributed Recipe Zoo (published 2025-08-04)

2025-07

• Kimi K2: Open Agentic Intelligence — Kimi K2: Open Agentic Intelligence (published 2025-07-28)
• Wan2.2: World's First Open-Source MoE Video Generation Model (launch announcement) — Wan2.2: World’s First Open-Source MoE Video Generation Model (launch announcement) (published 2025-07-28)
• Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decoding — Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decoding (published 2025-07-25)
• The Big LLM Architecture Comparison — The Big LLM Architecture Comparison (published 2025-07-19)
• QK-Clip: Taking Muon Further on the Scaleup Journey — QK-Clip: Taking Muon Further on the Scaleup Journey (published 2025-07-12)
• Kimi K2: Open Agentic Intelligence — Kimi K2: Open Agentic Intelligence (published 2025-07-11)

2025-06

• ERNIE 4.5 Technical Report — ERNIE 4.5 Technical Report (published 2025-06-29)
• Ming-Omni: A Unified Multimodal Model for Perception and Generation — Ming-Omni: A Unified Multimodal Model for Perception and Generation (published 2025-06-11)
• dots.llm1 Technical Report — dots.llm1 Technical Report (published 2025-06-06)

2025-05

• MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production — MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production (published 2025-05-16)

2025-01

• Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models — Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models (published 2025-01-21)

2024-01

• How To Scale — How To Scale (published 2024-01-22)

Unknown date

• Marin Selected Experiment Reports — Marin Selected Experiment Reports (published unknown)
• Qwen3-VL-30B-A3B-Instruct — Qwen3-VL-30B-A3B-Instruct (published unknown)
• Qwen3-VL-30B-A3B-Thinking — Qwen3-VL-30B-A3B-Thinking (published unknown)
• MiniMax-M2: a Mini model built for Max coding & agentic workflows — MiniMax-M2: a Mini model built for Max coding & agentic workflows (published unknown)