Slicing and Dicing MoEs — Margaret Li announcing a >2000-MoE-LM design-space study

TL;DR

A 9-tweet announcement from Margaret Li (@margs_li) of “Slicing and Dicing MoEs”: a study that trains over 2000 MoE language models to map the MoE design space — total vs active experts, expert size, shared experts, routing, and token dropping. The headline finding teased in tweet 1: “it’s all about expert size and count.” No arxiv link is visible in tweet 1; the underlying paper appears to be in flight rather than already posted. Filed as a tweet pointer; will be upgraded to the paper page when the arxiv id surfaces.

Key claims

Author claims >2000 MoE LMs were trained to disentangle the MoE design axes — total experts, active experts, expert size, shared experts, routing rule, token dropping [tweet 1/9].
The author’s one-line summary of the >2000-run sweep is that expert size and expert count dominate the other design axes [tweet 1/9].
The study explicitly lists five design axes — total vs active experts, expert size, shared experts, routing, token dropping — as the dimensions covered [tweet 1/9].

Method

Tweet announcement only. Tweet 1 of a 9-part thread; the remaining 8 tweets and any linked PDF / arxiv id are not visible in the fetched page (the embedded t.co shortlink resolves only back to the same status). The thread title ”📄🔪🍰 Slicing and Dicing MoEs” and the framing (“Tl;dr: it’s all about expert size and count”) indicate a forthcoming paper or report rather than a self-contained tweet artifact. Margaret Li is a known MoE researcher (Branch-Train-Merge, c-BTM lineage), so the claim of a 2000-run sweep is plausible but cannot be verified from the tweet alone.

Results

Not enumerated in tweet 1. Only the one-line takeaway (“it’s all about expert size and count”) is visible without the rest of the thread or the paper.

Why it’s interesting

Lands squarely on the MoE Routing Design concept page, which already aggregates nine 2026 papers picking apart specific axes — TopK vs soft gating (Routing-Free Mixture-of-Experts), router-parameter sharing across layers (Path-Constrained Mixture-of-Experts), load-balancing scope (Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models), attention-vs-experts FLOPs allocation (Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design), and quantile-based balancing (Quantile Balancing: A Hyperparameter-Free MoE Load Balancing Method). What none of those papers provides is a single sweep large enough to rank the axes against each other; “Slicing and Dicing MoEs,” if its scale claim holds, would be the first artifact in the cluster to do exactly that. The asserted ranking — expert size and count dominate — is also a direct empirical answer to the open question on MoE Routing Design about the unmapped multi-axis Pareto frontier, and it is consistent with the granularity-centric scaling-laws line (Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design §3, Fig. 5) which treats the experts-FLOPs share as the primary scaling axis.