VeOmni: Scaling Any Modality Model Training with Model-Centric Distributed Recipe Zoo

TL;DR

ByteDance’s training framework for any-to-any omni-modal LLMs. The central pitch is that prior multimodal trainers (DistMM, DistTrain, Optimus) tightly couple model definition with parallel logic and only target any-to-text, so adding a new modality encoder/decoder pair requires intrusive rewrites. VeOmni decouples the two via model-centric distributed recipes: FSDP/HSDP, DeepSpeed-Ulysses sequence parallelism (with an Async-Ulysses variant overlapping all-to-all with linear projections), and expert parallelism compose freely on a global device mesh, while modality-specific encoders/decoders plug in via a lightweight HF-style lm_encode / lm_embed / lm_generate / lm_head protocol. Reports >2,800 tokens/sec/GPU at 160K context on a 30B-A3B omni-modal MoE with 3D parallelism on 128 GPUs.

Key claims

Existing multimodal training frameworks (DistMM, DistTrain, Align Anything, Optimus) entangle parallel logic with model definition and target any-to-text only, leaving any-to-any (omni-modal) training without scalable infrastructure [§1, §2.2].
A plug-and-play encoder/decoder protocol — each modality module extends HuggingFace PreTrainedModel and implements lm_encode (training-time encode), lm_embed (inference-time embed), lm_generate (final generation), lm_head (decode foundation-model outputs) — is sufficient to register new modalities into a unified training/inference pipeline with minimal code change [§3.1].
FSDP / HSDP, DeepSpeed-Ulysses sequence parallelism, and expert parallelism are exposed via a high-level parallel plan API on a single global device mesh, so model code never imports a process group; switching FSDP→HSDP or composing 2D (FSDP+SP) → 3D (FSDP+SP+EP) is a config change rather than a model rewrite [§3.2, §3.2.4].
Async-Ulysses schedules the all-to-all communication required for sequence-parallel attention concurrently with linear-projection compute, claiming the latency hide yields “substantial” throughput gains [§3.2.2].
Operator-level communication/computation overlap is used for the all-to-all expert dispatch in MoE training, deliberately avoiding pipeline-level solutions (DualPipe-style) which the authors argue are sensitive to modality-specific load imbalance [§3.2.3].
On 8 GPUs with Qwen2-VL 7B, sequence parallelism extends supported context to 192K tokens at 61.5% MFU [§4.2, Fig. 4].
On 128 GPUs with Qwen2-VL 72B, sequence parallelism extends supported context to 96K tokens at 54.82% MFU [§4.2, Fig. 5].
On 128 GPUs with a 30B-A3B omni-modal MoE built on Qwen3-30B-A3B, 3D parallelism (FSDP+SP+EP) supports 160K context at >2,800 tokens/sec/GPU [§4.2, Fig. 6(a); Abstract].
Three structurally distinct omni-modal LLMs (Janus, LLaMA#Omni, Qwen3-MoE#Omni — combining SigLip/Qwen2.5-Omni-ViT/Qwen2.5-Omni-Whisper encoders with LLaMA/Qwen3-30B-A3B backbones and LlamaGen/MoVQGAN decoders) all show stable convergence across understanding (text/image/video/audio) and generation (text/image) tasks, separated by per-modality LM-loss and decoder-loss curves [§4.3, Fig. 6(b)].
System-level optimizations factored in alongside the parallelism stack: dynamic batching (sample packing under FlashAttention), liger-kernel + FlashAttention kernels, layer-wise recomputation with activation/optimizer offload, ByteCheckpoint for elastic distributed checkpointing extended to omni-modal heterogeneous components, and meta-device initialization for large models [§3.2.5].

Method

The framework has two halves. The model-side abstraction is a lightweight HuggingFace-compatible protocol: a foundation backbone in the middle, with arbitrary modality-specific encoders attached to its input embedding space and arbitrary decoders attached to its output, all implementing the same four lm_* methods. At training time, each encoder runs lm_encode to convert raw modality data into token embeddings that get inserted into the backbone’s input stream; each decoder runs lm_encode to provide training-time targets and lm_head to decode backbone outputs into the target modality. At inference, the backbone’s hidden states pass through the decoder’s lm_embed to produce next-step embeddings, and after the full token sequence is predicted, lm_generate runs the final modality-specific generation (e.g., diffusion sampling for image, vocoder for audio).

The systems-side abstraction is composable n-D parallelism on a global device mesh. FSDP (both FSDP1 and FSDP2 implementations are exposed under a unified API) and HSDP handle sharding parameters/gradients/optimizer states; DeepSpeed-Ulysses splits activations along the sequence dimension with all-to-all communications during attention, augmented with an Async-Ulysses variant that overlaps these all-to-alls with linear projections; expert parallelism shards MoE experts across devices with operator-level communication/computation overlap. The composition is configuration-driven — model code does not reference process groups or parallel state. The authors emphasize this is the same decoupling philosophy as TorchTitan/veScale on the LLM side, generalized to handle the heterogeneous encoder+backbone+decoder topology of omni-modal models.

Results

The headline omni-modal result is a 30B-A3B MoE omni-modal LLM (Qwen3-30B-A3B backbone, Qwen2.5-Omni encoders, MoVQGAN decoder) trained at >2,800 tokens/sec/GPU and 160K context on 128 GPUs under 3D parallelism. On dense Qwen2-VL backbones, the framework reaches 61.5% MFU at 192K context on Qwen2-VL-7B with 8 GPUs and 54.82% MFU at 96K context on Qwen2-VL-72B with 128 GPUs [§4.2]. Convergence is reported separately for LM-loss (text tokens) and decoder-loss (image tokens) across all three model architectures; all three show stable training curves across the union of text/image/video/audio understanding and text/image generation tasks [Fig. 6(b)]. The paper does not benchmark VeOmni head-to-head against Megatron-LM, TorchTitan, veScale, or DistTrain on a matched workload — comparisons are qualitative (decoupling, extensibility) rather than throughput-vs-baseline.

Why it’s interesting

Sits squarely on the Distributed training parallelism page alongside the other ByteDance-team filing veScale-FSDP: Flexible and High-Performance FSDP at Scale (veScale-FSDP) and Scaling and Optimizing Frontier Model Training (Fireworks AI) — same family of “decouple parallel logic from model definition, compose primitives per shape” arguments, but with the unique angle that the target shape is heterogeneous omni-modal (encoder + backbone + decoder, mixed dense/MoE, mixed continuous/discrete I/O) rather than uniform-shape MoE LLM training. For the Unified Multimodal Models page this is the systems-side counterpart to the architectural debate that page tracks (E2E vs Decoupled vs Agentic, MoT encoder-free, AR vs AR+Diffusion): VeOmni assumes the encoder/decoder/backbone decomposition and asks how to actually train such a thing at 30B-A3B-MoE / 160K-context scale. Complements MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production‘s drop-TP-for-SP+EP recipe — VeOmni stays with the same FSDP+SP+EP triple but adds the Async-Ulysses overlap and the omni-modal encoder/decoder protocol on top.