Skip to content
Bud's Wiki

Distributed training parallelism

TL;DR

Section titled “TL;DR”

Systems-level infrastructure for training large foundation models across many GPUs — the parallelism primitives (FSDP/ZeRO, tensor parallelism, pipeline parallelism, sequence/context parallelism, expert parallelism), the sharding formats they require, the schedulers that compose them, and the load balancers that handle non-uniform workloads. In late-2025/early-2026 the recurring claim is the same across LLM and DiT systems: the missing primitive is non-uniform — RaggedShard for non-element-wise optimizers and quantization (veScale-FSDP), true MPMD for heterogeneous pipeline stages (SpectraX), composable 4D parallelism for trillion-parameter MoE RL (Fireworks), and per-step dynamic sequence routing for variable-resolution video data (KnapFormer). Distinct from io-aware-kernel-design (which is the inner-loop kernel itself) and from training-stability-at-scale (which is the algorithmic-stability axis these systems must accommodate).

Key claims

Section titled “Key claims”

Recent contributions

Section titled “Recent contributions”
  • [2026-11-05] The Ultimate MoE Training Handbook: training Mixture-of-Experts models with Megatron-LM, setting by setting: Hugging Face’s Ultimate MoE Training Handbook centers on Megatron-LM as the production MoE training framework (citing Kimi-K2, Nemotron as deployment evidence) and walks from hardware characterization through dense baseline through single-node MoE to multi-node. Practitioner companion to the systems papers tracked here (Megatron-Core MoE, Fireworks, PithTrain, veScale-FSDP) and ships a MoE Memory Estimator for Megatron-LM HF Space that predicts GPU memory from a training command before launch.
  • [2026-11-05] ERNIE 5.5 Flash Base: ERNIE-5.5-Flash-Base contributes a distributed Muon over the expert-parallel ring recipe at 280B-MoE / 33.5T-token / FP8 scale: conventional single-ring dense-parameter synchronization across the whole cluster is replaced by routing dense-parameter updates through the same per-rank EP ring that MoE parameters use, with dense + MoE transfers combined into one collective — messages become larger (bandwidth more fully used), the pattern is more regular (fits EP communication), inter-node latency and jitter drop, and the optimizer’s updates stay exactly the same. Paired with inter-node memory-balanced sharding that spreads optimizer states over more GPUs rather than concentrating them on the first few (which naive Muon sharding does, causing OOM and slow checkpoint save/load/upload that lengthens failure recovery). A production answer to the “framework-default Muon lacks FSDP/HSDP support” open question — complements Training Dyna-2 at million-hour scale, repeatably Dyna-2’s topology-aware NVLink hybrid-sharding recipe with a different resolution of the same problem (co-opt EP ring vs duplicate above a node-count threshold).
  • [2026-11-04] GPU-accelerated Weight Teleportation: humans& ai’s follow-up to The 4-bitter Lesson: Balancing Stability and Performance in NVFP4 RL is a Blackwell-native delta weight sync pipeline that pushes the trainer↔rollout sync bottleneck to the hardware ceiling: distributed ownership-remapped GPU compression on all trainer ranks, double compression (GPU Snappy inner + CPU Zstd outer) with payload at 0.60% of model size, and heterogeneous decompression where DE consumes pinned CPU RAM directly into HBM, double-buffered-streamed one layer at a time with bitwise XOR on SMs. Reported Ampersand-config numbers (NVFP4 GLM-5.3, 64×B300): rollout pause 0.2196 s, end-to-end sync <4 s, no KV-cache eviction, auxiliary deltas (eval/fault-tolerance/resume) maintained against multiple base versions with distributed per-CPU memory. Authors claim intra-cluster P2P RDMA is now slower for most real workloads — a concrete endpoint for the delta-sync design trajectory slime adds lossless delta weight sync for Megatron ↔ SGLang RL disaggregation (PR #1806) Scaling and Optimizing Frontier Model Training (Fireworks AI) Composer 2 Technical Report opened.
  • [2026-10-21] Learning from Fleet Experience: Collective Experience Learning across 96 Robots: CEL/FATS (Tsinghua + AgiBot + Infinigence AI) is an operational robotics-side instance of distributed training infrastructure with a specific fault model this page doesn’t yet have: a 96-robot fleet producing training data while cloud inference workers serve the policy and training workers consume episodes, all over one shared network link. Three design choices worth filing: (i) HotSwarm hot-plugging gives each restarted worker a new identity so late in-flight actions addressed to a predecessor are rejected and incomplete episodes are discarded — fault-recovery discipline that keeps the training dataset consistent across control/inference/training worker failures; (ii) SwarmLink pools incoming observations across available cloud inference workers (batched request/reply) and streams execution records + outcomes to a trajectory collector that releases an episode to training only when all required fields and the outcome are present; (iii) interaction traffic takes priority over bulk transfers so cloud inference and bulk episode upload share the same link without blocking the collection loop. Reported scaling characteristic: under the same per-robot fault rate and downtime, the fraction of fleet-time available for collection stays steady as the fleet grows (each recovery still interrupts only the affected workers, not the whole fleet). Measured run: 9,834 episodes collected despite 30 fault incidents affecting 22 of 96 robots, 1.3% of total robot-time lost. Distinct from the GPU-centric parallelism recipes already filed by targeting the real-robot-fleet / cloud-inference-worker / training-worker triangle rather than inter-GPU communication, and complementary to Scale Robot Policy Evaluation with Ray (Distributed Sim-Eval on Anyscale) Anyscale/Ray’s reference architecture for sim-eval disaggregation on the evaluation side.
  • [2026-10-15] Frontier Pre-training at Microsoft AI (MAI-Base-2): MAI-Base-2 (Microsoft AI, 16K GB200, 2.8T/80B MoE) ships a resharding-based distributed Muon that squares the ZeRO-sharded-momentum vs. full-matrix-Newton-Schulz tension via per-step batched all-to-alls: FP32→BF16 shard update, parameter-shape-batched all-to-all into full matrices, symmetry-aware Newton–Schulz locally, second all-to-all back to shards and apply. Overlaps via multi-batch prefetching, caches resharding metadata (useful for sparse MoE with thousands of identically shaped parameters), and replicates (not shards) MoE router weights to skip the first all-to-all. Final cost: 2.1% / 165 ms per training iteration vs 55 ms theoretical communication-bound peak at 16K GPUs. Companion to Dyna-2’s topology-aware Muon sharding (Training Dyna-2 at million-hour scale, repeatably) and to the Dion variant (microsoft/dion — distributed Muon, Dion2, Dion, and NorMuon orthonormal optimizers) — three different frontier-scale answers to “how to distribute Muon’s full-matrix orthogonalization under FSDP/ZeRO”. Also contributes a multi-round dropless MoE that treats expert memory as a tunable fixed capacity rather than a padded worst-case, flipping MoE-in-every-layer from EGtime-suboptimal to optimal (24% → 35% MoE-layer MFU).
  • [2026-09-22] MiMo-V2.6: Scaling Reinforcement Learning Towards Self-Improvement: MiMo-V2.6 documents the infrastructure required to sustain 25K-sequence, 2.7–3.7B-token async RL batches at up to 1M context on a 1T MoE: decoupled control and data planes so trajectories carrying MoE routing + multimodal payloads move independently of orchestration, a pool of agent-harness frameworks for high-concurrency multi-framework rollout, a Sample Mixer to stabilize per-task sample composition across tasks with wildly different rollout durations and pass rates, plus partial rollout (Kimi Team 2025) amortized by large batch to keep rollout HBM-saturated. Reports 43.5% training / 43.8% rollout / 12.7% grader cost split (Pro).
  • [2026-09-13] Scaling Laws for Neural Language Models: Foundational back-fill: the Kaplan et al. LM scaling laws paper measures Bcrit(L)≈(2⋅108)/L5B_\mathrm{crit}(L) \approx (2 \cdot 10^8) / L^{5} tokens for Transformer LMs at convergence (~1M tokens for the largest models), giving the concrete numerical target that determines how large a data-parallel group can be before wasting compute.
  • [2026-09-13] An Empirical Model of Large-Batch Training: Foundational back-fill: the paper that first quantified when data-parallel batch scaling stops paying off. Critical batch size BcritB_\mathrm{crit} is the loss-dependent point beyond which doubling the batch size (and thus the number of data-parallel workers) yields <2× speedup in wall-clock training. Sets the theoretical ceiling for how many devices can profitably participate in a single data-parallel replica group before requiring hierarchical parallelism strategies.
  • [2026-09-12] AI Performance Engineering — curated resource list for GPU perf engineering and production inference: Wafer AI’s curated resource list; the Distributed Inference section catalogs parallelism/collectives/topology (Megatron-LM, NCCL, GB200 NVL, UALink 1.0, Ultra Ethernet 1.0.3), MoE serving (DeepSeek-V3, DeepEP, EPLB, MegaScale-Infer), and prefill/decode disaggregation (DistServe, Splitwise, Mooncake, NIXL, Dynamo) as an ordered reading path — inference-side companion to the training-side parallelism literature this concept tracks.
  • [2026-09-02] Training frontier knowledge work agents: A 397B RL training guide with SkyRL: Mercor’s 397B RL guide adds a concrete cluster-split recipe for fully-async agentic RL: 12:4 inference:train GPU node ratio for the 35B run, 12:8 for the 397B, tuned so SkyRL’s wait_for_generation_buffer panel stays near zero (trainer never idle). Rollout concurrency is bounded by min(systems ceiling = KV-cache capacity ÷ trajectory length, algorithmic ceiling = (max_staleness+1) × mini_batch × n_samples); Mercor set 550 (35B) and 300 (397B) — well under the 1024 algorithmic bound, so the systems ceiling bound. Train-inference logprob mismatch <0.03 without rollout router replay; the check caught a correctness issue in vLLM CPU offloading + GDN models + in-flight NCCL weight updates.
  • [2026-08-27] CARGO: Physical AI for Industrial Package Stacking: CARGO training distributes RL rollouts across “hundreds of NVIDIA GPU nodes” via Ray for a Sim2Real robotics task-planning policy — a filed industrial-deployment datapoint on Ray-orchestrated distributed rollout collection at scale.
  • [2026-08-17] Training Dyna-2 at million-hour scale, repeatably: Dyna Robotics’s infrastructure post-mortem for the 1M-hour Dyna-2 pre-training run introduces a topology-aware Muon sharding recipe that picks the strategy from node count: world-sharding (each rank updates every Nth parameter, all-reduce afterward) at small N, FSDP-Hybrid (split work over NVLink inside the node via per-node NCCL sub-groups, duplicate the work across nodes) at large N. Reports ~3× speedup at scale and a mean-vs-median analysis showing fully-sharded pays 2.6× median on mean step time due to 7.6× broadcast traffic on InfiniBand vs NVLink. Concrete production answer to the “framework-default Muon lacks distributed support” open question the microsoft/dion — distributed Muon, Dion2, Dion, and NorMuon orthonormal optimizers line has been circling, and an instance of the page’s “non-uniformity as first-class primitive” thesis at the optimizer-sharding layer rather than the FSDP/PP/EP/CP layer.
  • [2026-08-12] MAGI-2 Preview: Scaling Video Generation Models Efficiently: Sand.ai’s Hierarchical Head Parallel maps the two bandwidth tiers of a training cluster onto the two moving parts of Multi-Head MoE: inter-node InfiniBand exchanges fixed-shape head-activation slices (send/receive counts derivable statically from the head partition, buffers preallocatable), intra-node NVLink handles ZeRO/FSDP-style sharding of expert weights + gradients + optimizer states with on-demand materialization and reshard-after-computation. Sibling to MegaScale-MoE’s SP+EP-drop-TP recipe and Megatron-Core’s Parallel Folding, but resolves the attention-vs-MoE parallelism mismatch by pre-regularizing the MoE communication shape via head partitioning rather than by rearranging the parallelism groups.
  • [2026-08-01] MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training: MOPD decouples per-domain RL teacher production from capability integration, making Stage-2 teachers fully parallelisable (independent recipes, hyperparameters, and restart cost); teacher prefill is deployed as an asynchronous service outside the trainer so teacher cost hides behind student sampling.
  • [2026-07-24] Open Dreamer — Open-source Dreamer 4 world-model implementation in JAX/Flax: Open Dreamer is a full-stack world-model training pipeline in JAX/Flax NNX with ArrayRecord + Grain data loading and Orbax checkpointing — a rare filed instance of an end-to-end JAX WM training codebase (prior JAX-side entries have been infrastructure libraries: SpectraX for MPMD pipeline parallelism, Jasmine as the cited prior).
  • [2026-07-14] Accelerating the ImageNet Moment of Embodied AI: RLinf Brings a 25× System Optimization to BEHAVIOR: RLinf’s BEHAVIOR optimization extends the “non-uniformity as a first-class primitive” thesis from the trainer side to the environment side of embodied RL rollouts. A hybrid-partition pipeline parallel refuses pure process-level stage assignment (which stalls on the serial physics segment inside each vectorized env) and instead partitions env slices inside vectorized envs, mapping same-stage slices across all subprocesses; exposed as a single env.pipeline_stage_num knob. The physics simulation frequency is adjusted with stage count so physical time scale stays consistent — a domain-specific correctness constraint the LLM-side pipeline systems (SpectraX, DualPipeV, streaming Fireworks) don’t have to solve. Sibling to the async-RL rollout coordinators (AReaL StalenessManager, Fireworks streaming pipeline) but on the env-producer side rather than the trainer-consumer side.
  • [2026-07-10] The 4-bitter Lesson: Balancing Stability and Performance in NVFP4 RL: Bit-exact numerical contract between trainer (TransformerEngine) and sampler (FlashInfer) as a distributed-systems requirement for NVFP4 RL: quantized values, per-block FP8 scales, per-token FP32 scales, and Four-Over-Six error accumulation must be bitwise identical regardless of weight layout or swizzling, otherwise the trainer/sampler divergence reopens under RL rollouts. First open-source implementation of this contract across a real Megatron ↔ SGLang RL stack.
  • [2026-07-01] Scale Robot Policy Evaluation with Ray (Distributed Sim-Eval on Anyscale): Anyscale’s post applies the same async producer/consumer decoupling pattern this page tracks on the training side (Fireworks streaming pipeline, AReaL StalenessManager, KnapFormer compute bags) to the evaluation side: many-simulator producers and few-policy-replica consumers, connected by HTTP with action-chunk amortization across the hop; runtime_env propagates gated-model credentials uniformly to Serve replicas and simulator subprocess workers, an operational detail training-infra systems don’t have to solve because they don’t span gated HF weights + sandboxed sim workers the way policy-eval does.
  • [2026-06-29] Demystifying NVSHMEM: A System-Level Analysis on Symmetric Memory and Device-Initiated Operations in GPU Communication: Systems-level walkthrough of NVSHMEM — the OpenSHMEM-based PGAS library underpinning GPU-initiated, one-sided communication via symmetric memory — that sits at the comm-runtime layer beneath FSDP/PP/EP systems on this page; DeepEP is used as the case study showing where device-initiated issue beats CPU-orchestrated MPI/NCCL on fine-grained MoE all-to-all.
  • [2026-06-23] PithTrain: A Compact and Agent-Native MoE Training System: PithTrain (CMU/MLC, Lai et al.): a ~11K-line pure-Python MoE training framework that composes the standard 4D parallelism axes (FSDP / Pipeline via DualPipeV from DeepSeek-V3 / Context / Expert) plus FP8 training and DCP checkpointing, and matches Megatron-LM training throughput on 4 of 5 MoE configurations (GPT-OSS-20B, Qwen3-30B-A3B, DeepSeek-V2-Lite) on H100/B200 — at 1/13th the LoC of Megatron-LM (149K) and 1/15th of DeepSpeed (167K). The novel framing is that codebase size and structure (compactness, Python-native, no plugin/registry indirection) are themselves a load-bearing axis once AI coding agents are the framework’s primary maintainers; introduces “agent-task efficiency” (ATE) as a metric complementary to throughput. Sits beside Fireworks at the opposite scope/agent-cost point on this page’s tradeoff curve.
  • [2026-05-28] DiffusionBlocks: Block-wise Neural Network Training via Diffusion Interpretation: DiffusionBlocks (Sakana AI, ICLR 2026) introduces a block-independent training primitive orthogonal to the four standard parallelism axes (FSDP / PP / CP / EP). By reinterpreting residual blocks as Euler steps of the reverse PF-ODE and assigning each block its own noise-level range, K blocks can be trained without any inter-block gradient communication — K-fold activation-memory reduction with zero sync overhead. Distinct from pipeline parallelism (which still requires forward/backward passes across stages) and from FSDP (which still requires end-to-end backprop). Open whether DiffusionBlocks composes with the per-stage / per-shard non-uniformity primitives this page tracks; no head-to-head numbers exist yet.
  • [2026-05-26] FaaScale: Unlocking Fast LLM Scaling for Serverless Inference: FaaScale (MLSys 2026) ports pipeline parallelism to the serving side: PipeCast forms transient cross-node inference pipelines dynamically during model-weight multicast over RDMA/GDR, so a set of nodes that collectively hold all model blocks can start inference before any single node holds the full model. Same “non-uniformity is the missing primitive” thesis as veScale-FSDP / SpectraX / KnapFormer — here the dynamic dimension is which subset of nodes forms a pipeline at any given moment during a scale-out, with nodes leaving the cross-node pipeline and switching to local execution as soon as they finish receiving the full model. Extends the wiki’s pipeline-parallelism coverage from training infra into serverless inference bootstrap.
  • [2026-05-25] lmms-engine — A simple, unified multimodal models training engine (LMMs-Lab, v0.1 Oct 2025): research-scale PyTorch reference engine that composes the same FSDP2 + Ulysses-SP + EP triple as VeOmni, adds Muon (MuonWithAdam with hardcoded exclusion list + dim==2 filter), Liger fused kernels (CrossEntropy/RMSNorm/RoPE/SwiGLU, claimed 30% memory reduction), first-fit sequence packing with unpadded Flash Attention (claimed 20-25%→35-40% MFU on Qwen2.5-VL SFT), and Native Sparse Attention (BAGEL only). Notable scope: ships first-class trainers for diffusion-language models (dllm_trainer), WanVideo flow-matching, RAE adversarial+EMA, and SiT alongside the standard HF VLM trainer — broader out-of-the-box example matrix than any other system on this page. Settles into a different niche than the frontier-scale Fireworks stack and the JAX-MPMD SpectraX path: cheap-to-fork PyTorch reference for the FSDP2 + USP + EP recipe.
  • [2026-05-25] AReaL Code Walk Through — SGLang RL team's deep dive into Yi Wu's asynchronous RL framework (SGLang RL team): code walkthrough of Yi Wu’s AReaL framework, which is the rollout-side counterpart to the trainer-side systems on this page. Two contributions are filed-worthy as primitives: (1) StalenessManager.get_capacity gives a closed-form formula for how many rollouts can safely be in flight given a max_staleness future-version budget, decoupling rollout production from trainer consumption without violating staleness contracts; (2) DistRolloutCoordinator.redistribute uses FFD on token counts to re-bin completions across data-parallel ranks, generalizing the KnapFormer compute-bag idea from DiT sequence parallelism to async-RL data parallelism. Together with Scaling and Optimizing Frontier Model Training (Fireworks AI)‘s streaming pipeline schedule (trainer side), these are the two filed pieces that address asynchronous data arrival as a first-class scheduling problem.
  • [2026-05-24] VeOmni: Scaling Any Modality Model Training with Model-Centric Distributed Recipe Zoo (ByteDance, Aug 2025): VeOmni — model-centric distributed training framework for omni-modal (any-to-any) LLMs. Pairs FSDP/HSDP + Ulysses-SP (with Async-Ulysses overlap) + EP on a single global device mesh, with a plug-and-play HuggingFace-compatible encoder/decoder protocol (lm_encode / lm_embed / lm_generate / lm_head) so adding new modality modules requires no parallel-logic changes. Reports 61.5% MFU at 192K context on Qwen2-VL-7B (8 GPUs), 54.82% MFU at 96K context on Qwen2-VL-72B (128 GPUs), and >2,800 tokens/sec/GPU at 160K context on a 30B-A3B omni-modal MoE built on Qwen3-30B-A3B (128 GPUs, FSDP+SP+EP). Same ByteDance group as veScale-FSDP; extends the “non-uniformity is the missing primitive” thesis from shard granularity to model topology.
  • [2026-05-24] KnapFormer: An Online Load Balancer for Efficient Diffusion Transformers Training (Adobe Research, Kai Zhang et al.): introduces compute bags as the unit of dynamic sequence-parallel grouping on top of Ulysses, with a knapsack-style allocator that re-routes sequence chunks across bags per step to equalize work under highly heterogeneous data (mixed images, keyframes, videos at varying res/fps). MMDiT/Flux integration scoped to three small surfaces in the forward pass. First filed system that treats per-step input-shape variability as the load-balancing target — sibling LLM systems on this page assume roughly uniform input shapes and balance optimizer/comm buffers instead.
  • [2026-05-24] SpectraX — True MPMD Pipeline Parallelism for JAX (Erfan Zare Chavoshi): the JAX-side counterpart of veScale-FSDP. True MPMD pipeline parallelism via sxjit, compiling a distinct XLA program per rank with stages of differing class/shape; ships nine named schedules including DualPipeV (DeepSeek-V3) and KimiK2 as first-class objects with analytical bubble/memory metrics. Production-deployed as the NN core under EasyDeL (77 model families).
  • [2026-05-24] Scaling and Optimizing Frontier Model Training (Fireworks AI) (Fireworks AI): productized trainer stack behind Cursor’s Composer 2 and now Kimi K2.5, Qwen3.5 397B, MiniMax M2.5. Composes 4D parallelism (FSDP/PP/CP/EP) per training shape on PyTorch + GB200, with MXFP8 native grouped-GEMM expert kernels, FA4 MLA shapes, per-RL-algorithm fused-loss kernels (~2× for GRPO), FP8 QAT, streaming pipeline schedule for async RL data, and LoRA-side tricks (frozen-expert quantization, optimizer offload, multi-session adapters). Trains a 1T+ MoE at >1M context.
  • [2026-05-23] veScale-FSDP: Flexible and High-Performance FSDP at Scale (ByteDance): the foundational filing for this concept. Identifies that the FSDP-side missing primitive is non-uniform shard granularity — RaggedShard supports arbitrary atomic block shapes and arbitrary per-device block counts, generalizing all prior FSDP sharding formats. Backs grouped tensors with a DBuffer primitive for zero-copy aligned collectives. 1166% throughput / 1630% memory wins over best baseline; native support for Muon, Shampoo, 8-bit Adam, block-wise FP8 without intrusive model code.

Open questions

Section titled “Open questions”
  • The five filed papers don’t all measure on the same axis: veScale-FSDP reports throughput/memory at 1024-GPU LLaMA-3-70B / internal MoE, Fireworks reports validated 1T-context 1M-token GB200 training, SpectraX reports CPU microbenchmarks + 77-family EasyDeL production deployment, KnapFormer is the only one targeting DiT (no quantitative results in the README), and VeOmni targets omni-modal MoE at 128 GPUs / 30B-A3B / 160K context without head-to-head Megatron/TorchTitan numbers. A direct head-to-head on a single workload (e.g. a 50B+ MoE pretraining run at fixed FLOPs) doesn’t exist yet.
  • KnapFormer’s compute-bag idea is sequence-parallel-side; veScale-FSDP’s RaggedShard is data-parallel-side; SpectraX is pipeline-parallel-side; VeOmni is model-topology-side; AReaL’s FFD redistributor is rollout-side. The natural question is whether these five primitives compose — a system where shard granularity, pipeline stage class, sequence-parallel group, encoder/decoder module, and rollout assignment are all per-step or per-shape dynamic. None of the filings prototype this.
  • For DiT pretraining specifically, the wiki has multiple algorithmic-side efficiency recipes (Diffusion training efficiency tracks REPA, SRA, Self-Flow, UL, Foveated Diffusion) but only KnapFormer on the systems side. What’s the systems-side analog of REPA’s >10× convergence speedup? Is it the imbalanced-bag-stall gap, the activation-checkpointing cost, the AllReduce overhead, or something else?
  • The streaming pipeline schedule for asynchronous RL data (Fireworks), AReaL’s producer-consumer rollout coordinator with staleness/capacity bounds, and the per-step routing plan for heterogeneous DiT data (KnapFormer) are structurally similar problems: all three decouple the trainer’s schedule from a non-batched data arrival pattern. Is there a unified scheduling abstraction or are these genuinely separate problems?
  • How much of veScale-FSDP’s RaggedShard win is from MoE-specific padding elimination vs. from the DBuffer primitive (zero-copy collectives) vs. from the planner heuristic? The headline 566% / 1630% number bundles all three; the individual contributions aren’t ablated.
  • SpectraX is JAX-only; veScale-FSDP / Fireworks / KnapFormer / VeOmni are PyTorch. Is the MPMD vs. SPMD-with-shard-map distinction (SpectraX’s central argument) actually decisive at trillion-parameter scale, or does PyTorch’s per-rank flexibility (via fully_shard + custom routing) close the gap in practice?
  • For omni-modal training, is the right comm/compute overlap target attention’s all-to-all (Async-Ulysses) or MoE expert dispatch (operator-level EP overlap)? VeOmni does both but doesn’t ablate which carries the throughput. The Fireworks stack relies on pipeline-level streaming for RL; VeOmni explicitly refuses pipeline-level overlap on modality-imbalance grounds. Whether the operator-level vs pipeline-level choice generalizes beyond ByteDance’s specific workloads is open.
  • Does the omni-modal encoder/decoder plug-in API generalize to non-HuggingFace stacks? VeOmni’s protocol assumes PreTrainedModel extension; a JAX/Flax port (e.g., on top of SpectraX) is not described.
  • Is AReaL’s closed-form staleness/capacity formula correct under heterogeneous workflows? The formula assumes one trainer-version-per-batch and uniform per-task cost; multi-turn workflows or workflows that take radically different wall-times per sample (the MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling case) may need a different bound that the walkthrough doesn’t characterize.
  • Where’s the line between research-scale (lmms-engine: ~8-128 GPU, single-mesh FSDP2 + USP + EP, hackable Python) and frontier-scale (Fireworks: 1T MoE / 1M context / GB200 / 4D parallelism)? Both stacks use the same primitives but differ by an order of magnitude in scale and in how much custom kernel work is bundled. The transition is presumably smooth but unmeasured.

Papers

Section titled “Papers”

2026-04

Section titled “2026-04”

2026-02

Section titled “2026-02”

2025-12

Section titled “2025-12”

2025-11

Section titled “2025-11”

2025-10

Section titled “2025-10”

2025-09

Section titled “2025-09”

2025-08

Section titled “2025-08”

2025-01

Section titled “2025-01”

Unknown date

Section titled “Unknown date”