Enabling Up to 41% Faster Pre-training: MXFP8 and DeepEP for DeepSeek-V3 on B200 with TorchTitan

TL;DR

A PyTorch + Nebius joint study reports end-to-end pre-training of DeepSeek-V3 (16B and 671B MoE variants) on a 256× NVIDIA B200 cluster using TorchTitan, isolating two orthogonal optimizations: MXFP8 quantization (via TorchAO) on grouped GEMMs and DeepEP for expert-parallel all-to-all communication. On the 671B model with EP=32, DeepEP alone delivers +32% throughput (651 → 859 tokens/sec) and combining it with MXFP8-on-grouped-GEMMs hits +41% (918 tokens/sec). On the 16B model, MXFP8 vs BF16 loss curves are visually indistinguishable over 1,500 steps. Full Slurm scripts and TorchTitan configs are published in the Nebius ML-CookBook. Useful as a concrete reference point for what’s currently achievable on Blackwell with open-source tooling.

Key claims

DeepEP replaces NCCL’s generic all-to-all with hierarchical NVLink+RDMA kernels purpose-built for MoE dispatch/combine and delivers +32% throughput on DeepSeek-V3 671B at EP=32 across 32 nodes [§Experiment 1].
MXFP8 on grouped GEMMs composes additively with DeepEP because the two optimizations target independent bottlenecks (compute vs communication), reaching +41% combined throughput on the 671B [§Experiment 1, Figure 2].
MXFP8 training converges equivalently to BF16 over 1,500 steps on the 16B MoE variant, with loss curves “virtually identical” including the zoomed-in last 30% [§Experiment 2, Figs. 3-4].
MXFP8 assigns a shared E8M0 exponent per block of 32 elements rather than a single scale per tensor, and Blackwell’s 5th-gen tensor cores execute the resulting GEMMs natively via tcgen05.mma at full FP8 throughput [§Background].
The MXFP8-on-grouped-GEMMs recipe pays off when GEMMs are large (compute scales O(GMNK), quantization overhead scales O(MK) or O(GNK)); on smaller MoE models the overhead can match or exceed the speedup, yielding flat-to-negative performance [§Background].
DeepEP’s gains come from four design choices: pipelined NVLink → RDMA-over-InfiniBand transfer, GPU-initiated RDMA via NVSHMEM + IBGDA (no CPU involvement), fused metadata transfer bundling token embeddings + expert indices + routing weights into a single op, and configurable SM allocation for communication kernels [§Background].
The 671B run does not yet use MXFP8 on the linear (attention / shared-expert) layers because torch.compile does not currently support MXFP8 linear + TP, and the 671B config requires TP=2; additional throughput is therefore expected once that lands [§Future works].
671B configuration uses TP=2, PP=2, EP=32, ETP=1, FSDP=auto, full activation checkpointing, sequence length 8192, local batch 64, LR 1e-4 with 2000 warmup steps, C4 dataset [§Experiment 1].

Method

The experiments compose three open-source stacks on top of Nebius’s B200 cluster: TorchTitan as the training framework, TorchAO as the MXFP8 quantization layer (converting nn.Linear and torch._grouped_mm ops to dynamically quantized MXFP8 variants with BF16 accumulation), and DeepEP as a replacement all-to-all backend for MoE dispatch/combine. The “MXFP8 GMM” recipe wraps grouped GEMMs in TorchAO’s _to_mxfp8_then_scaled_grouped_mm autograd function, which quantizes inputs to MXFP8 just before each of the three GEMMs (forward output, input gradient, weight gradient). DeepEP plugs into TorchTitan’s expert-parallel pipeline at EP=32 spanning 32 nodes, using NVSHMEM’s symmetric memory + IBGDA for GPU-initiated RDMA. The whole pipeline runs under torch.compile for model and loss.

Three configurations were evaluated on 671B: BF16 baseline with NCCL all-to-all, BF16 + DeepEP, and MXFP8 GMM + DeepEP. The 16B convergence study compared BF16 vs MXFP8 GMM both under standard EP. Cluster: 32 nodes × 8 B200 GPUs (256 total), NVLink/NVSwitch intra-node, InfiniBand inter-node, Soperator (Slurm-on-Kubernetes) scheduling with continuous NCCL all-reduce health checks.

Results

DeepSeek-V3 671B throughput, EP=32, 256 B200s [§Summary table]:
- BF16 + standard EP: 651 tokens/sec (baseline)
- BF16 + DeepEP: 859 tokens/sec (+32.0%)
- MXFP8 GMM + DeepEP: 918 tokens/sec (+41.0%)
Loss convergence (16B MoE, 1500 steps): BF16 and MXFP8 curves are “virtually identical” with no meaningful divergence even in the final 30% of training [§Experiment 2, Figs. 3-4].
Where the gains come from: DeepEP delivers the single largest individual lever (+32%) because inter-node all-to-all dominates step time at EP=32; MXFP8 GMM adds +9 percentage points on top because it accelerates expert grouped GEMMs while leaving the (already-shrunk) communication budget alone.
Caveat on coverage: MXFP8 is applied only to grouped GEMMs in the 671B run; attention and shared-expert linears remain BF16 because of the torch.compile + TP=2 + MXFP8-linear incompatibility. So +41% is a floor, not a ceiling, for what MXFP8 + DeepEP will eventually deliver on this configuration.
All recipes, Slurm scripts, and TorchTitan configs are published at the Nebius ml-cookbook repository for reproducibility.

Why it’s interesting

This is the most concrete public throughput number to date for open-source MXFP8 + DeepEP training of a 671B-parameter DeepSeek-V3 on commodity B200 clusters, and it pairs naturally with Scalable Training of Mixture-of-Experts Models with Megatron Core — NVIDIA’s Megatron-Core MoE systems paper, which advertises 1233/1048 TFLOPS/GPU on GB300/GB200 for the same model but in a closed framework; this PyTorch + Nebius writeup is the open-stack equivalent and provides the head-to-head delta between BF16 baseline, DeepEP-only, and MXFP8+DeepEP for a community-reproducible config. The “+32% from DeepEP alone” number is a useful empirical pin on the Megatron-Core paper’s qualitative claim that inter-node all-to-all “can consume up to 60% of training time” at EP=32.

Complements veScale-FSDP: Flexible and High-Performance FSDP at Scale (ByteDance’s MoE-aware FSDP variant) by sitting at the orthogonal layer: veScale-FSDP fixes parameter sharding under FSDP2-style even-padding inflation; this report fixes the communication backend for MoE all-to-all. The two would stack cleanly. Also a useful data point for Training stability at scale‘s low-precision sub-story (Gated Attention / SSO both claim outlier-suppression has low-precision-friendliness as a side-effect) — MXFP8’s per-block E8M0 scaling on DeepSeek-V3 grouped GEMMs converges with no degradation, suggesting the architecture is already low-precision-friendly enough without explicit anti-outlier interventions.