Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data

TL;DR

Jha, Li, Leskovec, Liang & Zettlemoyer study how Mixture-of-Experts models degrade under training-data repetition and find that MoEs overfit substantially faster than dense counterparts. Sweeping 80M–1B active (up to 8.5B total) params, they show 80M dense models tolerate ≥8× data repetition with minimal degradation while MoEs already begin to suffer at 4× and, after 32×, drop below the dense baseline they otherwise outperform on unique data. The overfitting rate scales with total rather than active parameters — i.e. with sparsity. Strong masking-based regularization (dropout) can push MoEs back ahead of dense even at 64× repetition, but no method fully recovers all-unique-data performance. Mechanistically, MoE routing stabilizes very early in training and post-stabilization expert specialization correlates with overfitting on repeated data.

Key claims

At 80M active params, dense models tolerate ≥8× repetition with minimal degradation, while matched-active-param MoEs begin degrading at 4× and fall below dense after 32× repeats [Abstract, §Results].
Repetition-induced degradation grows with sparsity and is governed by total rather than active parameter count — the standard “capacity” story that active-params matter for compute-optimality does not carry over to the overfitting axis [Abstract].
Masking-based regularization (dropout in particular) partially rescues MoEs: with strong dropout, MoEs outperform dense baselines even at >64× repetition, though no method matches all-unique-data performance [Abstract, §Regularization].
MoE router decisions stabilize very early in training (well before overfitting sets in) and the degree of expert specialization at that stabilized routing correlates with overfitting severity on repeated data [Abstract, §Analysis].
The result holds across single- and multi-domain data mixes and across MoE variants (varying expert count and granularity) up to 1B active / 8.5B total params [Abstract].

Method

Controlled sweep across three axes: (a) data-repetition rate on both single-domain and multi-domain training mixes, (b) MoE design (expert count, expert granularity, sparsity ratio), and (c) model scale from 80M active / ~800M-total up to 1B active / 8.5B total. Matched-compute dense baselines are trained at each active-parameter point. For each configuration the training corpus is repeated a controlled number of times and validation loss is tracked; degradation is measured relative to the all-unique-data reference at the same total token budget. A second phase re-runs the sweep with standard regularizers layered on (dropout, weight decay, expert-choice variants) to test whether the overfitting is fundamental or a training-recipe artifact. Internal analysis instruments the router: tracks when the per-token expert assignment distribution stops moving during training, and measures per-expert domain/topic concentration at convergence.

Results

Headline pattern: at 80M active parameters and matched compute, dense models absorb 8× data repetition with minimal validation-loss increase; MoEs at the same active-param setting are already degrading at 4× and after 32× they lose their unique-data advantage over dense and continue to worsen. Overfitting rate scales with sparsity: at fixed active params, a higher total-param MoE (more experts or wider experts, hence higher sparsity ratio) degrades faster than a lower-total-param MoE. Strong dropout inverts the story past ~64× repetition — regularized MoEs beat dense — but the all-unique-data configuration remains the best absolute setting, so regularization is a mitigation, not a fix. Routing analysis: post-router-stabilization (which happens very early — well before overfitting is measurable in validation loss), the degree of per-expert specialization is a proxy for how much the model will overfit under further repetition.

Why it’s interesting

Directly relevant to the wiki’s live thread on MoE robustness at scale. Sits alongside Scaling Domain Data Repetition in LLM Pretraining (Li et al. — how many repeats each domain benefits from on dense-adjacent models at fixed TPP) and Internal Data Repetition Destroys Language Models (Kazdan et al. — worst-case repetition structure on dense LMs). This paper adds the missing sparsity axis both were silent on: at the same active-parameter budget, MoEs sit on a strictly worse point of the data-repetition tradeoff than dense, and the crossover with the dense baseline arrives inside the practical 32×–64× range that frontier training already targets.

Refines the routing-design vocabulary on MoE Routing Design. That page’s live thread on router behavior — Mitigate Silent Expert Death In Ultra-Sparse MoE on silent expert death, Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models on LBL-scope-hides-specialization, Quantile Balancing: A Hyperparameter-Free MoE Load Balancing Method on aux-free bias overriding router decisions — has repeatedly shown that standard training metrics are blind to what the router is actually doing. This paper adds that the specialization pattern established at early router stabilization is what determines downstream overfitting to repeated data, which suggests specialization-vs-diversity is a routing-design axis distinct from load balance. It also connects to Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization‘s general point that MoE at scale is a different optimization regime from dense.

Practical implication for Luma-adjacent work: if a video-DiT or WAM run is planning to repeat a limited high-quality domain slice (as LingBot-Video: Scaling Mixture-of-Experts Video Pretraining for Embodied Intelligence and MAGI-2 Preview: Scaling Video Generation Models Efficiently both use fine-grained MoE), the dense-baseline repetition budget is not a safe transfer — MoEs likely need a smaller repetition count or a masking-based regularizer to reach the same effective ceiling.