Distributed training parallelism

TL;DR

Systems-level infrastructure for training large foundation models across many GPUs — the parallelism primitives (FSDP/ZeRO, tensor parallelism, pipeline parallelism, sequence/context parallelism, expert parallelism), the sharding formats they require, the schedulers that compose them, and the load balancers that handle non-uniform workloads. In late-2025/early-2026 the recurring claim is the same across LLM and DiT systems: the missing primitive is non-uniform — RaggedShard for non-element-wise optimizers and quantization (veScale-FSDP), true MPMD for heterogeneous pipeline stages (SpectraX), composable 4D parallelism for trillion-parameter MoE RL (Fireworks), and per-step dynamic sequence routing for variable-resolution video data (KnapFormer). Distinct from io-aware-kernel-design (which is the inner-loop kernel itself) and from training-stability-at-scale (which is the algorithmic-stability axis these systems must accommodate).

Key claims

• The dominant theme across late-2025/early-2026 distributed training systems is non-uniformity as a first-class primitive: shard granularity (veScale-FSDP RaggedShard), per-stage compilation (SpectraX MPMD), per-shape parallelism composition (Fireworks 4D), and per-step sequence routing (KnapFormer compute bags) all reject the “uniform shape across all ranks” assumption that earlier systems baked in (veScale-FSDP: Flexible and High-Performance FSDP at Scale §4; SpectraX — True MPMD Pipeline Parallelism for JAX README “MPMD Runtime”; Scaling and Optimizing Frontier Model Training (Fireworks AI) §“Composable 4D parallelism”; KnapFormer: An Online Load Balancer for Efficient Diffusion Transformers Training README “Description”). VeOmni extends the same theme to heterogeneous model topologies (encoder + backbone + decoder, mixed dense/MoE, mixed continuous/discrete modalities) via a model-centric API that decouples parallel logic from model definition (VeOmni: Scaling Any Modality Model Training with Model-Centric Distributed Recipe Zoo §3.1, §3.2). AReaL extends it again to the async-RL rollout side: per-rank token-count load balancing via FFD on variable-length GRPO completions (AReaL Code Walk Through — SGLang RL team's deep dive into Yi Wu's asynchronous RL framework §“DistRolloutCoordinator”).
• The four standard parallelism axes for trillion-parameter MoE training are FSDP / Pipeline / Context / Expert, composed per “training shape” rather than chosen globally; a dense 8B model may use only FSDP while a 1T MoE at 256K context uses all four (Scaling and Optimizing Frontier Model Training (Fireworks AI) §“Composable 4D parallelism”). For omni-modal MoE, the equivalent composition is FSDP + Ulysses-SP + EP on a single global device mesh, with the SP all-to-all overlapped with linear projections via Async-Ulysses (VeOmni: Scaling Any Modality Model Training with Model-Centric Distributed Recipe Zoo §3.2.2, §3.2.4). The same FSDP2 + USP + EP triple, plus Muon and Liger fused kernels, is the default stack in the research-scale lmms-engine — A simple, unified multimodal models training engine reference engine.
• Element-wise and row-wise FSDP sharding cannot align shard boundaries with the atomic blocks required by Muon/Shampoo (full 2D matrix locality) or by block-wise FP8 quantization, forcing intrusive model rewrites; RaggedShard with customizable atomic block shape generalizes all prior FSDP sharding formats as special cases (veScale-FSDP: Flexible and High-Performance FSDP at Scale §2.3, §4, Fig. 4).
• FSDP2’s per-parameter even-sharding pays a 14% iteration-time penalty (interleaved AllGather copy-out + ReduceScatter copy-in); Megatron-FSDP’s Stride(0) constraint inflates the comm buffer by 33% on MoE due to padding; veScale-FSDP claims 1166% throughput over best baseline on internal MoE on 1024 GPUs (veScale-FSDP: Flexible and High-Performance FSDP at Scale §6.1, Table 1).
• True MPMD pipeline parallelism compiles a distinct XLA program per rank, so stages can differ in class, parameter shape, and I/O shape — heterogeneous pipelines (embed → blocks → head, or vision-tower + text-tower) are first-class rather than padded to a common shape (SpectraX — True MPMD Pipeline Parallelism for JAX README “MPMD Runtime”, “Stage regions”).
• Nine pipeline schedules with different bubble-vs-memory-vs-transport tradeoffs ship behind a single Schedule interface, exposing analytical bubble_ratio, peak_activations, total_steps for pre-launch comparison: GPipe, Std1F1B, Eager1F1B, ZeroBubbleH1, InterleavedH1/GPipe/1F1B+1, DualPipeV (DeepSeek-V3), KimiK2 (SpectraX — True MPMD Pipeline Parallelism for JAX README “Supported schedules”).
• Streaming pipeline schedules redesigned for RL’s asynchronous data arrival give up to an order-of-magnitude improvement in first-result latency at low input-QPS-to-batch-size ratios, with exact gradient parity to batched accumulation (Scaling and Optimizing Frontier Model Training (Fireworks AI) §“Streaming pipeline schedule”). The rollout-side analog: AReaL’s WorkflowExecutor decouples trainer from inference via a producer-consumer queue gated by an explicit staleness/concurrency budget capacity = min(concurrency_limit, (max_staleness + version + 1) * batch_size − current_samples) (AReaL Code Walk Through — SGLang RL team's deep dive into Yi Wu's asynchronous RL framework §“StalenessManager”).
• For DiT pretraining the dominant per-step waste is not per-GPU FLOPs but imbalanced bag stalls — Ulysses-style sequence parallelism with statically-chosen groups leaves some ranks idle for seconds when batches mix 256² images with 1080p video; dynamic per-step bag assignment recovers the gap (KnapFormer: An Online Load Balancer for Efficient Diffusion Transformers Training README “Description”, animation). The async-RL analog of the same problem is per-rank token-count imbalance under variable-length completions, which AReaL solves with FFD-based redistribution after all_gather_tensor_container (AReaL Code Walk Through — SGLang RL team's deep dive into Yi Wu's asynchronous RL framework §“DistRolloutCoordinator”).
• Optimizer-state CPU/GPU offload reduces peak GPU memory by >40% on a Qwen3-30B MoE / 8×H200 setup with bit-identical training results, opening single-node training for what previously required multi-node (Scaling and Optimizing Frontier Model Training (Fireworks AI) §“Optimizer state offloading”).
• Frozen-expert low-precision storage with on-the-fly bf16 dequant during forward is what makes LoRA on 1T-class MoE single-node-feasible — Kimi K2.5’s 384 experts fit on a single 8-GPU node only because experts are stored at ~4× compression and dequantized inside the tensor-core MMA (Scaling and Optimizing Frontier Model Training (Fireworks AI) §“Low-precision expert quantization”).
• Numerical-correctness validation at MoE scale is non-trivial: Qwen3.5-35B alone required solving 9 distinct gradient-correctness bugs across shared experts, router gates, GQA, and DeltaNet layers before training matched expected loss curves (Scaling and Optimizing Frontier Model Training (Fireworks AI) §“Numerical parity”).
• For omni-modal training the comm/compute mismatch shows up at attention’s all-to-all (Ulysses-SP) rather than at MoE expert dispatch; VeOmni’s Async-Ulysses overlaps the SP all-to-all with linear projections, and refuses pipeline-level MoE overlap (DualPipe) on the basis that pipeline scheduling is rigid under modality-specific load imbalance — preferring operator-level overlap instead (VeOmni: Scaling Any Modality Model Training with Model-Centric Distributed Recipe Zoo §3.2.2, §3.2.3).
• Sequence packing with use_rmpad (true unpadded Flash Attention) is claimed to lift Qwen2.5-VL fine-tuning MFU from 20-25% to 35-40% in the lmms-engine — A simple, unified multimodal models training engine reference engine — same first-fit-bin-packing scheme as VeOmni but exposed as a single packing_strategy: first_fit + use_rmpad: true YAML pair.

Recent contributions

• [2026-07-14] Accelerating the ImageNet Moment of Embodied AI: RLinf Brings a 25× System Optimization to BEHAVIOR: RLinf’s BEHAVIOR optimization extends the “non-uniformity as a first-class primitive” thesis from the trainer side to the environment side of embodied RL rollouts. A hybrid-partition pipeline parallel refuses pure process-level stage assignment (which stalls on the serial physics segment inside each vectorized env) and instead partitions env slices inside vectorized envs, mapping same-stage slices across all subprocesses; exposed as a single env.pipeline_stage_num knob. The physics simulation frequency is adjusted with stage count so physical time scale stays consistent — a domain-specific correctness constraint the LLM-side pipeline systems (SpectraX, DualPipeV, streaming Fireworks) don’t have to solve. Sibling to the async-RL rollout coordinators (AReaL StalenessManager, Fireworks streaming pipeline) but on the env-producer side rather than the trainer-consumer side.
• [2026-07-10] The 4-bitter Lesson: Balancing Stability and Performance in NVFP4 RL: Bit-exact numerical contract between trainer (TransformerEngine) and sampler (FlashInfer) as a distributed-systems requirement for NVFP4 RL: quantized values, per-block FP8 scales, per-token FP32 scales, and Four-Over-Six error accumulation must be bitwise identical regardless of weight layout or swizzling, otherwise the trainer/sampler divergence reopens under RL rollouts. First open-source implementation of this contract across a real Megatron ↔ SGLang RL stack.
• [2026-07-01] Scale Robot Policy Evaluation with Ray (Distributed Sim-Eval on Anyscale): Anyscale’s post applies the same async producer/consumer decoupling pattern this page tracks on the training side (Fireworks streaming pipeline, AReaL StalenessManager, KnapFormer compute bags) to the evaluation side: many-simulator producers and few-policy-replica consumers, connected by HTTP with action-chunk amortization across the hop; runtime_env propagates gated-model credentials uniformly to Serve replicas and simulator subprocess workers, an operational detail training-infra systems don’t have to solve because they don’t span gated HF weights + sandboxed sim workers the way policy-eval does.
• [2026-06-29] Demystifying NVSHMEM: A System-Level Analysis on Symmetric Memory and Device-Initiated Operations in GPU Communication: Systems-level walkthrough of NVSHMEM — the OpenSHMEM-based PGAS library underpinning GPU-initiated, one-sided communication via symmetric memory — that sits at the comm-runtime layer beneath FSDP/PP/EP systems on this page; DeepEP is used as the case study showing where device-initiated issue beats CPU-orchestrated MPI/NCCL on fine-grained MoE all-to-all.
• [2026-06-23] PithTrain: A Compact and Agent-Native MoE Training System: PithTrain (CMU/MLC, Lai et al.): a ~11K-line pure-Python MoE training framework that composes the standard 4D parallelism axes (FSDP / Pipeline via DualPipeV from DeepSeek-V3 / Context / Expert) plus FP8 training and DCP checkpointing, and matches Megatron-LM training throughput on 4 of 5 MoE configurations (GPT-OSS-20B, Qwen3-30B-A3B, DeepSeek-V2-Lite) on H100/B200 — at 1/13th the LoC of Megatron-LM (149K) and 1/15th of DeepSpeed (167K). The novel framing is that codebase size and structure (compactness, Python-native, no plugin/registry indirection) are themselves a load-bearing axis once AI coding agents are the framework’s primary maintainers; introduces “agent-task efficiency” (ATE) as a metric complementary to throughput. Sits beside Fireworks at the opposite scope/agent-cost point on this page’s tradeoff curve.
• [2026-05-28] DiffusionBlocks: Block-wise Neural Network Training via Diffusion Interpretation: DiffusionBlocks (Sakana AI, ICLR 2026) introduces a block-independent training primitive orthogonal to the four standard parallelism axes (FSDP / PP / CP / EP). By reinterpreting residual blocks as Euler steps of the reverse PF-ODE and assigning each block its own noise-level range, K blocks can be trained without any inter-block gradient communication — K-fold activation-memory reduction with zero sync overhead. Distinct from pipeline parallelism (which still requires forward/backward passes across stages) and from FSDP (which still requires end-to-end backprop). Open whether DiffusionBlocks composes with the per-stage / per-shard non-uniformity primitives this page tracks; no head-to-head numbers exist yet.
• [2026-05-26] FaaScale: Unlocking Fast LLM Scaling for Serverless Inference: FaaScale (MLSys 2026) ports pipeline parallelism to the serving side: PipeCast forms transient cross-node inference pipelines dynamically during model-weight multicast over RDMA/GDR, so a set of nodes that collectively hold all model blocks can start inference before any single node holds the full model. Same “non-uniformity is the missing primitive” thesis as veScale-FSDP / SpectraX / KnapFormer — here the dynamic dimension is which subset of nodes forms a pipeline at any given moment during a scale-out, with nodes leaving the cross-node pipeline and switching to local execution as soon as they finish receiving the full model. Extends the wiki’s pipeline-parallelism coverage from training infra into serverless inference bootstrap.
• [2026-05-25] lmms-engine — A simple, unified multimodal models training engine (LMMs-Lab, v0.1 Oct 2025): research-scale PyTorch reference engine that composes the same FSDP2 + Ulysses-SP + EP triple as VeOmni, adds Muon (MuonWithAdam with hardcoded exclusion list + dim==2 filter), Liger fused kernels (CrossEntropy/RMSNorm/RoPE/SwiGLU, claimed 30% memory reduction), first-fit sequence packing with unpadded Flash Attention (claimed 20-25%→35-40% MFU on Qwen2.5-VL SFT), and Native Sparse Attention (BAGEL only). Notable scope: ships first-class trainers for diffusion-language models (dllm_trainer), WanVideo flow-matching, RAE adversarial+EMA, and SiT alongside the standard HF VLM trainer — broader out-of-the-box example matrix than any other system on this page. Settles into a different niche than the frontier-scale Fireworks stack and the JAX-MPMD SpectraX path: cheap-to-fork PyTorch reference for the FSDP2 + USP + EP recipe.
• [2026-05-25] AReaL Code Walk Through — SGLang RL team's deep dive into Yi Wu's asynchronous RL framework (SGLang RL team): code walkthrough of Yi Wu’s AReaL framework, which is the rollout-side counterpart to the trainer-side systems on this page. Two contributions are filed-worthy as primitives: (1) StalenessManager.get_capacity gives a closed-form formula for how many rollouts can safely be in flight given a max_staleness future-version budget, decoupling rollout production from trainer consumption without violating staleness contracts; (2) DistRolloutCoordinator.redistribute uses FFD on token counts to re-bin completions across data-parallel ranks, generalizing the KnapFormer compute-bag idea from DiT sequence parallelism to async-RL data parallelism. Together with Scaling and Optimizing Frontier Model Training (Fireworks AI)‘s streaming pipeline schedule (trainer side), these are the two filed pieces that address asynchronous data arrival as a first-class scheduling problem.
• [2026-05-24] VeOmni: Scaling Any Modality Model Training with Model-Centric Distributed Recipe Zoo (ByteDance, Aug 2025): VeOmni — model-centric distributed training framework for omni-modal (any-to-any) LLMs. Pairs FSDP/HSDP + Ulysses-SP (with Async-Ulysses overlap) + EP on a single global device mesh, with a plug-and-play HuggingFace-compatible encoder/decoder protocol (lm_encode / lm_embed / lm_generate / lm_head) so adding new modality modules requires no parallel-logic changes. Reports 61.5% MFU at 192K context on Qwen2-VL-7B (8 GPUs), 54.82% MFU at 96K context on Qwen2-VL-72B (128 GPUs), and >2,800 tokens/sec/GPU at 160K context on a 30B-A3B omni-modal MoE built on Qwen3-30B-A3B (128 GPUs, FSDP+SP+EP). Same ByteDance group as veScale-FSDP; extends the “non-uniformity is the missing primitive” thesis from shard granularity to model topology.
• [2026-05-24] KnapFormer: An Online Load Balancer for Efficient Diffusion Transformers Training (Adobe Research, Kai Zhang et al.): introduces compute bags as the unit of dynamic sequence-parallel grouping on top of Ulysses, with a knapsack-style allocator that re-routes sequence chunks across bags per step to equalize work under highly heterogeneous data (mixed images, keyframes, videos at varying res/fps). MMDiT/Flux integration scoped to three small surfaces in the forward pass. First filed system that treats per-step input-shape variability as the load-balancing target — sibling LLM systems on this page assume roughly uniform input shapes and balance optimizer/comm buffers instead.
• [2026-05-24] SpectraX — True MPMD Pipeline Parallelism for JAX (Erfan Zare Chavoshi): the JAX-side counterpart of veScale-FSDP. True MPMD pipeline parallelism via sxjit, compiling a distinct XLA program per rank with stages of differing class/shape; ships nine named schedules including DualPipeV (DeepSeek-V3) and KimiK2 as first-class objects with analytical bubble/memory metrics. Production-deployed as the NN core under EasyDeL (77 model families).
• [2026-05-24] Scaling and Optimizing Frontier Model Training (Fireworks AI) (Fireworks AI): productized trainer stack behind Cursor’s Composer 2 and now Kimi K2.5, Qwen3.5 397B, MiniMax M2.5. Composes 4D parallelism (FSDP/PP/CP/EP) per training shape on PyTorch + GB200, with MXFP8 native grouped-GEMM expert kernels, FA4 MLA shapes, per-RL-algorithm fused-loss kernels (~2× for GRPO), FP8 QAT, streaming pipeline schedule for async RL data, and LoRA-side tricks (frozen-expert quantization, optimizer offload, multi-session adapters). Trains a 1T+ MoE at >1M context.
• [2026-05-23] veScale-FSDP: Flexible and High-Performance FSDP at Scale (ByteDance): the foundational filing for this concept. Identifies that the FSDP-side missing primitive is non-uniform shard granularity — RaggedShard supports arbitrary atomic block shapes and arbitrary per-device block counts, generalizing all prior FSDP sharding formats. Backs grouped tensors with a DBuffer primitive for zero-copy aligned collectives. 1166% throughput / 1630% memory wins over best baseline; native support for Muon, Shampoo, 8-bit Adam, block-wise FP8 without intrusive model code.

Open questions

• The five filed papers don’t all measure on the same axis: veScale-FSDP reports throughput/memory at 1024-GPU LLaMA-3-70B / internal MoE, Fireworks reports validated 1T-context 1M-token GB200 training, SpectraX reports CPU microbenchmarks + 77-family EasyDeL production deployment, KnapFormer is the only one targeting DiT (no quantitative results in the README), and VeOmni targets omni-modal MoE at 128 GPUs / 30B-A3B / 160K context without head-to-head Megatron/TorchTitan numbers. A direct head-to-head on a single workload (e.g. a 50B+ MoE pretraining run at fixed FLOPs) doesn’t exist yet.
• KnapFormer’s compute-bag idea is sequence-parallel-side; veScale-FSDP’s RaggedShard is data-parallel-side; SpectraX is pipeline-parallel-side; VeOmni is model-topology-side; AReaL’s FFD redistributor is rollout-side. The natural question is whether these five primitives compose — a system where shard granularity, pipeline stage class, sequence-parallel group, encoder/decoder module, and rollout assignment are all per-step or per-shape dynamic. None of the filings prototype this.
• For DiT pretraining specifically, the wiki has multiple algorithmic-side efficiency recipes (Diffusion training efficiency tracks REPA, SRA, Self-Flow, UL, Foveated Diffusion) but only KnapFormer on the systems side. What’s the systems-side analog of REPA’s >10× convergence speedup? Is it the imbalanced-bag-stall gap, the activation-checkpointing cost, the AllReduce overhead, or something else?
• The streaming pipeline schedule for asynchronous RL data (Fireworks), AReaL’s producer-consumer rollout coordinator with staleness/capacity bounds, and the per-step routing plan for heterogeneous DiT data (KnapFormer) are structurally similar problems: all three decouple the trainer’s schedule from a non-batched data arrival pattern. Is there a unified scheduling abstraction or are these genuinely separate problems?
• How much of veScale-FSDP’s RaggedShard win is from MoE-specific padding elimination vs. from the DBuffer primitive (zero-copy collectives) vs. from the planner heuristic? The headline 566% / 1630% number bundles all three; the individual contributions aren’t ablated.
• SpectraX is JAX-only; veScale-FSDP / Fireworks / KnapFormer / VeOmni are PyTorch. Is the MPMD vs. SPMD-with-shard-map distinction (SpectraX’s central argument) actually decisive at trillion-parameter scale, or does PyTorch’s per-rank flexibility (via fully_shard + custom routing) close the gap in practice?
• For omni-modal training, is the right comm/compute overlap target attention’s all-to-all (Async-Ulysses) or MoE expert dispatch (operator-level EP overlap)? VeOmni does both but doesn’t ablate which carries the throughput. The Fireworks stack relies on pipeline-level streaming for RL; VeOmni explicitly refuses pipeline-level overlap on modality-imbalance grounds. Whether the operator-level vs pipeline-level choice generalizes beyond ByteDance’s specific workloads is open.
• Does the omni-modal encoder/decoder plug-in API generalize to non-HuggingFace stacks? VeOmni’s protocol assumes PreTrainedModel extension; a JAX/Flax port (e.g., on top of SpectraX) is not described.
• Is AReaL’s closed-form staleness/capacity formula correct under heterogeneous workflows? The formula assumes one trainer-version-per-batch and uniform per-task cost; multi-turn workflows or workflows that take radically different wall-times per sample (the MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling case) may need a different bound that the walkthrough doesn’t characterize.
• Where’s the line between research-scale (lmms-engine: ~8-128 GPU, single-mesh FSDP2 + USP + EP, hackable Python) and frontier-scale (Fireworks: 1T MoE / 1M context / GB200 / 4D parallelism)? Both stacks use the same primitives but differ by an order of magnitude in scale and in how much custom kernel work is bundled. The transition is presumably smooth but unmeasured.

Papers

2026-04

• Scaling and Optimizing Frontier Model Training (Fireworks AI) — Scaling and Optimizing Frontier Model Training (Fireworks AI) (published 2026-04-03)

2026-02

• veScale-FSDP: Flexible and High-Performance FSDP at Scale — veScale-FSDP: Flexible and High-Performance FSDP at Scale (published 2026-02-25)

2025-12

• SpecBundle & SpecForge v0.2: Production-Ready Speculative Decoding Models and Framework — SpecBundle & SpecForge v0.2: Production-Ready Speculative Decoding Models and Framework (published 2025-12-23)
• Kling-Omni Technical Report — Kling-Omni Technical Report (published 2025-12-18)
• The Trinity Manifesto — Arcee's open-weight MoE family (Mini + Nano + Large roadmap) — The Trinity Manifesto — Arcee’s open-weight MoE family (Mini + Nano + Large roadmap) (published 2025-12-01)

2025-11

• Loads and Loads of Fluffy Kittens: Compute-Communication Kernels with Multi-GPU ThunderKittens — Loads and Loads of Fluffy Kittens: Compute-Communication Kernels with Multi-GPU ThunderKittens (published 2025-11-17)
• MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling — MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling (published 2025-11-14)
• The MoE 101 Guide: From Theory to Production (Daria Soboleva, Cerebras) — The MoE 101 Guide: From Theory to Production (Daria Soboleva, Cerebras) (published 2025-11-14)

2025-10

• Introducing PyTorch Monarch — Introducing PyTorch Monarch (published 2025-10-22)
• Paris: A Decentralized Trained Open-Weight Diffusion Model — Paris: A Decentralized Trained Open-Weight Diffusion Model (published 2025-10-03)
• lmms-engine — A simple, unified multimodal models training engine — lmms-engine — A simple, unified multimodal models training engine (published 2025-10-01)

2025-09

• LongCat-Flash Technical Report — LongCat-Flash Technical Report (published 2025-09-01)

2025-08

• VeOmni: Scaling Any Modality Model Training with Model-Centric Distributed Recipe Zoo — VeOmni: Scaling Any Modality Model Training with Model-Centric Distributed Recipe Zoo (published 2025-08-04)

2025-01

• KnapFormer: An Online Load Balancer for Efficient Diffusion Transformers Training — KnapFormer: An Online Load Balancer for Efficient Diffusion Transformers Training (published 2025-01-01)

Unknown date

• Marin Selected Experiment Reports — Marin Selected Experiment Reports (published unknown)
• SpectraX — True MPMD Pipeline Parallelism for JAX — SpectraX — True MPMD Pipeline Parallelism for JAX (published unknown)
• AReaL Code Walk Through — SGLang RL team's deep dive into Yi Wu's asynchronous RL framework — AReaL Code Walk Through — SGLang RL team’s deep dive into Yi Wu’s asynchronous RL framework (published unknown)