MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production

TL;DR

ByteDance Seed’s production system for training large MoE models, deployed in their datacenters. The headline argument: as hardware FLOPs outpace interconnect bandwidth, intra-node Tensor Parallelism becomes a liability for MoE — its all-gather/reduce-scatter on the critical path can exceed the self-attention compute it parallelizes. MegaScale-MoE drops TP entirely and instead pairs Pipeline Parallelism (inter-node) with two intra-node strategies tailored per layer-component: DeepSpeed-Ulysses-style Sequence Parallelism for attention, and Expert Parallelism for FFNs. On top, it adds fine-grained inter- and intra-operator communication–computation overlap and lower-precision (BF16/FP8) parameter synchronization. Reports 1.88× higher MFU than Megatron-LM training a 352B-parameter MoE on 1,440 Hopper GPUs.

Key claims

Communication is the dominant bottleneck in production MoE training — 43.6% of forward-pass time and 32% of total training time on a ByteDance internal MoE on Hopper GPUs, and naively extending TP across nodes pushes this past 50% [§1].
TP partitions the expert hidden dimension and degrades GEMM efficiency on already-fine-grained expert matrices; EP keeps experts full-size on each device and replaces TP for the FFN [§3.2].
For attention, DeepSpeed-Ulysses Sequence Parallelism reduces critical-path communication to roughly one-quarter of TP’s on an NVL8 workstation, because it sends activations only along sequence/head dims rather than all-gathering/reduce-scattering each layer [§3.1, Eq. 1–2].
Context Parallelism (sequence-sharding) is rejected for production MoE because causal-masking imbalance under CP — even with zigzag interleaving — forces the pipeline to wait on the most imbalanced batch, hurting end-to-end throughput [§3.1].
For top-k > 6, replacing all-to-all token dispatch with all-gather + local scatter + reduce-scatter is faster than vanilla all-to-all, because the ring-pattern collectives only talk to NVLink neighbors while all-to-all is N-to-N [§3.2, Fig. 7].
The auxiliary balance loss is computed per-GPU-group (treating co-located experts as a single virtual expert), à la DeepSeek-V2, rather than per individual expert [§3.2].
Fine-grained intra-operator overlap fuses tile-level communication into GEMM and GroupedGEMM kernels via in-device-memory barriers, eliminating host-side stream control overhead and exploiting GPU copy engines so all SMs remain on compute [§4.2].
Selective activation rematerialization stores only ~half the activations and hides the recomputation behind communication on the backward critical path, so memory drops with no speed regression [§4.1, Fig. 8].
BF16 mixed-precision training can drop inter-node parameter synchronization from FP32 to BF16, halving that channel’s volume; FP8 training replaces BF16 reduce-scatter with FP8 communication with FP32 reduction and tailored quantization for convergence [§5].
End-to-end result: 1.41M tokens/sec on a 352B-parameter MoE across 1,440 Hopper GPUs, 1.88× higher MFU than Megatron-LM [Abstract, §6].

Method

The design space is constructed by stripping TP from the standard 5-axis parallelism layout (DP / TP / SP / PP / EP) and re-allocating its role: PP handles inter-node parameter sharding; intra-node, attention runs Ulysses-style SP (heads sharded across the NVL8 domain, with two all-to-alls per attention block to swap between sequence-sharded and head-sharded layouts), and FFN runs EP (each expert replicated across one device, with all-to-all or all-gather + reduce-scatter for token routing depending on top-k). The router treats the per-node group of experts as one virtual expert for load-balance computation.

The communication–computation overlap stack has three layers. (1) Inter-operator: each MoE layer’s forward and backward passes are decomposed into per-operator GPU kernels rather than relying on torch.autograd, giving a hand-scheduled MoELayer macro that reorders communication and computation across the dependency graph (e.g. recomputing ffn_in from RMSNorm + all-gather behind the FC2 GroupedGEMM backward). (2) Intra-operator with GEMMs: A2A+GEMM and GEMM+A2A fused kernels use device-memory barriers for tile-level signalling, plus swizzling to avoid NVLink contention. (3) Intra-operator with GroupedGEMMs: AG+scatter+GroupedGEMM and GroupedGEMM+gather+RS kernels reorder tokens along the sequence dimension by routed-expert index and then by source rank, so each compute tile depends on tokens from the minimum number of remote ranks.

Results

1.88× MFU improvement over Megatron-LM on a 352B-parameter MoE / 1,440 NVIDIA Hopper GPUs, 1.41M tokens/s training throughput [Abstract, §6].
Communication share before optimization: 43.6% of forward pass, 32% of total training; multi-node TP cases observed at >50% [§1].
For grouped-query attention, Ulysses-SP attention communication is roughly 1/4 of TP-attention on NVL8 (Eq. 1 vs Eq. 2 with GQA ratio).
All-gather-based EP beats all-to-all-based EP for Mixtral-8×7B once top-k > 6 [Fig. 7].
Selective rematerialization stores half the activations with no measurable slowdown when overlapped with backward-pass communication [§4.1].
BF16 inter-node sync halves parameter-synchronization communication overhead vs FP32 sync under BF16 mixed-precision training [§5].

Why it’s interesting

This is the algorithmic-systems counterpart to Scalable Training of Mixture-of-Experts Models with Megatron Core (NVIDIA’s Megatron-Core MoE report): same problem (the attention-prefers-TP/CP vs MoE-prefers-EP mismatch at scale), different resolution. Megatron-Core resolves it via MoE Parallel Folding — keeping TP/CP for attention but decoupling MoE-layer parallelism so it can independently use EP with ETP=1. MegaScale-MoE resolves it by eliminating TP altogether and replacing it with Ulysses SP for attention. Both ship to production; the Jiaming/sand-ai conversation note in the pointer suggests the SP-not-CP choice is itself contested at very long context lengths, which is exactly the regime where ring attention (Context Parallelism with overlapped KV passing) is usually argued for.

The paper is also a sibling to veScale-FSDP: Flexible and High-Performance FSDP at Scale (ByteDance’s own veScale-FSDP / RaggedShard) — both are ByteDance Seed systems reports addressing MoE-at-scale, but at different layers: veScale-FSDP fixes parameter sharding (the RaggedShard format that handles uneven expert counts); MegaScale-MoE fixes the communication schedule between shards. The auxiliary-balance-per-GPU-group recipe also touches the design space catalogued in MoE Routing Design — specifically the “scope over which $f_i$ is computed” axis that Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models showed silently determines expert specialization.