Quantile Balancing: A Hyperparameter-Free MoE Load Balancing Method

TL;DR

Jianlin Su (RoFormer / RoPE / MuonClip) proposes Quantile Balancing (QB), a hyperparameter-free, auxiliary-loss-free load balancing scheme for top-K MoE routing. QB formulates the per-step expert-assignment problem as a constrained linear program — given each token’s routing scores, find the per-expert bias offsets that would have produced exactly balanced load on the previous step — and solves it via alternating quantile iterations. Unlike DeepSeek’s loss-free bias update (which nudges biases by a fixed ±0.001 each step) or GShard-style auxiliary losses (which require tuning a balancing coefficient), QB has no hyperparameters and converges to balance faster, especially in the chaotic warmup phase. The Marin team validated the method at 32B-A5B / 1e22 FLOPs / 326B tokens with zero loss spikes and is now scaling to 1e23 FLOPs.

Key claims

QB casts MoE load balancing as a constrained linear program: solve for the per-expert bias terms that, when added to last step’s routing scores, would have produced uniform load across all experts [Jianlin’s blog].
The LP is solved by an alternating quantile-based iteration — for each token find the activation threshold, then for each expert find the bias that activates a balanced number of tokens [Jianlin’s blog; Open Athena §Quantile load balancing].
QB has zero hyperparameters: no balancing coefficient, no fixed bias increment, no momentum term [Jianlin’s blog].
At 32B-A5B / 1e22 FLOPs / 326B tokens, QB produced zero loss spikes and required no leading dense layers, no aux loss, and no capacity overload factor [Open Athena §Quantile load balancing].
Compared to the Marin team’s prior 1e21 run (fixed-increment bias + small aux loss + 2 leading dense layers), QB balanced experts significantly faster in the first 5,000 steps, with no extended dead expert windows [Open Athena §Quantile load balancing].
The current QB implementation is a one-step approximation based on the prior step’s routing scores, leaving room for an exact-LP variant in future work [Open Athena §Quantile load balancing].

Method

QB targets the standard top-K MoE setup: each token $t$ produces routing scores $s_t \in \mathbb{R}^E$ over $E$ experts and the top-K experts are activated. Existing balancers either (a) add an auxiliary balancing loss weighted by $\alpha$ (GShard), or (b) maintain a learnable bias $b_e$ per expert updated by $\pm u$ each step based on expert load (DeepSeek “loss-free”; refined by Arcee’s SMEBU with momentum). All require hyperparameter tuning that varies by layer, model size, and loss regime.

QB instead asks: given the scores from the previous step, what bias vector $b \in \mathbb{R}^E$ would have produced exactly balanced top-K assignment? This is a constrained linear program. QB approximates its solution with a two-step alternating procedure: (1) for each token, compute the activation threshold (a quantile over the K-th largest score in $s_t + b$ ); (2) for each expert, find the bias adjustment that brings its load to the balanced target. Iterating these two steps converges to the LP solution; the current Marin implementation uses a single iteration per step as a fast approximation.

Because the bias only enters the top-K selection (not the gating weights used downstream), routing-vector geometry is left to free-form optimization. Marin’s run shows expert routing vectors first converging around step 4,000 then spreading out into distinct subspaces by step 60,000, suggesting QB does not force premature specialization.

Results

Marin’s validation run is 32B total / 5B active parameters, 64 routed experts, top-4, 32 layers, 1,600 neurons per expert (~3M total expert neurons), trained on 326B tokens (1e22 FLOPs). Key empirical findings from the Open Athena writeup:

Zero loss spikes across the entire 326B-token run.
No supplementary techniques required: no leading dense layers, no auxiliary loss, no capacity overload factor.
Faster warmup balancing vs the prior 1e21 fixed-increment-bias + aux-loss + 2-dense-leading-layer recipe, which had experts dead for 50+ steps in the first 5,000 steps.
One residual issue: layer 4 has a pair of experts persistently at ~70% of balanced load — not degenerate, possibly natural early-layer token-space asymmetry.

A 1e23-FLOP run (Marin’s “current hero” MoE) is underway with QB; Percy Liang’s tweet flags QB as the reason they have stable training at that scale.

Why it’s interesting

Load balancing is one of the most tuning-sensitive parts of MoE training, and a hyperparameter-free recipe that holds up at 1e22+ FLOPs is a real quality-of-life and a real risk reduction for any team training MoEs from scratch. The framing — “solve the LP that would have balanced last step” — is cleaner than the patchwork of aux losses, bias-update magnitudes, and momentum coefficients in current production recipes; it also makes the balancer’s behavior auditable (you can inspect the per-step LP solution rather than tuning a black-box knob). For Luma’s interest in long-stable training of large models, QB sits next to MuonClip / qk-clip as another “remove a tuning axis entirely” trick from the same author. The independent BIP-Based Balancing work (Sun 2025) suggests this LP-style framing is becoming a small genre.