Skip to content
Bud's Wiki

lmms-engine — A simple, unified multimodal models training engine

TL;DR

Section titled “TL;DR”

LMMs-Engine (v0.1, Oct 2025) is LMMs-Lab’s open-source, lean training engine aimed at unified multimodal models at scale. It composes FSDP2, Tensor Parallelism, Ulysses Sequence Parallelism, Expert Parallelism, Muon, Liger fused kernels, first-fit sequence packing, and Native Sparse Attention behind a YAML-configured runner, and ships ready-to-run examples for 20+ architectures — Qwen2.5/3-VL, Qwen3-VL MoE, Qwen3-Omni MoE, BAGEL, WanVideo, dLLM (Qwen3), RAE-SigLip, and SiT among them. Positioning is intentional: a hackable middle ground between Megatron / nanotron / VeOmni at the heavy end and Axolotl / LLaMA-Factory at the SFT end.

Key claims

Section titled “Key claims”
  • The example matrix advertises a single feature-set table across 13 model families, with FSDP2 enabled in every row and Muon enabled in every row; Tensor Parallelism is only marked supported for Qwen3-VL [README §Featured Examples].
  • Expert Parallelism (EP) is supported for the three MoE rows (Qwen3-VL MoE, Qwen3-MoE, Qwen3-Omni MoE) and is the only sharding mode marked specifically for MoE in the table [README §Featured Examples].
  • Sequence packing uses first-fit bin packing and is claimed to lift Qwen2.5-VL finetuning MFU from “20-25%” to “35-40%” peak when combined with use_rmpad (unpadded Flash Attention) [README §Featured Examples, §Optimizations].
  • Liger Kernel integration provides Triton-fused CrossEntropy/RMSNorm/RoPE/SwiGLU with a claimed “30% memory reduction” headline and is enabled by a single use_liger_kernel: true flag, applied at runtime via a monkey-patch system rather than model edits [README §Optimizations, §Liger Kernel block].
  • Muon is integrated as MuonWithAdam — Newton-Schulz orthogonalization with Triton kernels, distributed via DTensor (FSDP2-compatible), with a hardcoded exclusion list (emb, norm, lm_head, bias, wte, wpe, output, a_proj, b_proj, conv1d, rotary) plus a dimension==2 filter selecting which params get the Muon update vs. AdamW [README §Muon Optimizer block].
  • Native Sparse Attention is wired only for the BAGEL family at present, exposed as a monkey_patch: type: nsa block with configurable block size, compress type (weightedpool/linear/avgpool), kernel size/stride, topk, init/local blocks, and sliding-window size [README §Native Sparse Attention block, §Featured Examples].
  • The trainer registry exposes five distinct trainer classes — hf_trainer, dllm_trainer, wan_trainer, rae_trainer, sit_trainer — so diffusion language models, WanVideo flow-matching, RAE adversarial+EMA, and SiT interpolant training all share the FSDP2/Muon/Liger plumbing but keep loss-specific logic isolated [README §Supported Trainers].
  • The included Qwen3-VL example targets native-resolution VLM training with sequences “10K+ tokens” via Ulysses SP, and is the only row in the matrix with both TP and USP enabled [README §Featured Examples row “Qwen3-VL”].

Method

Section titled “Method”

The engine is a YAML-configured runner: TrainRunner(config).build().run(), with build() doing lazy initialization of model → dataset+processor → monkey-patch application → trainer (HF/DLLM/Wan/RAE/SiT) → execution. Extension points are factory-pattern registries (@register_dataset, @register_processor, @register_model), so adding a custom dataset, processor, or model is in-place rather than via fork. Distributed launch uses torchrun natively (or accelerate launch --use_fsdp) — the design preference is for PyTorch-native FSDP2 over the legacy FSDP path.

Parallelism composes via a single mesh: FSDP2 (DTensor-based, per-parameter even-sharding, with reshard_after_forward and activation checkpointing as YAML flags), Tensor Parallelism, and Ulysses Sequence Parallel (sp_ulysses_degree) for ultra-long contexts. Optimizations layer on top: Flash Attention with use_rmpad for true unpadding, sequence packing with packing_strategy: first_fit and packing_length, Liger-fused kernels applied via monkey-patch, Muon as a distributed DTensor optimizer, and NSA as an opt-in patch (BAGEL only). The “monkey patch system” (configs under lmms_engine/configs/monkey_patch/) lets per-model kernel substitutions ship without modifying upstream model code.

Results

Section titled “Results”

There are no headline benchmark numbers in the README itself — the engine refers out to a separate MFU reference doc for Model FLOPs Utilization across configurations, and to its Docker images for pre-built environments. The headline claims in-text are: packing + use_rmpad lifts Qwen2.5-VL MFU into the 35-40% range (vs. 20-25% without), and Liger kernels claim 30% memory reduction. Qwen3-VL is called out as the example with TP × USP both enabled, targeting 10K+ token vision sequences. Downstream projects that report training on LMMs-Engine are LongVT (long-video tool-calling reasoning) and OpenMMReasoner (multimodal reasoning recipe).

Why it’s interesting

Section titled “Why it’s interesting”

Sits in the same productized-trainer slot as VeOmni: Scaling Any Modality Model Training with Model-Centric Distributed Recipe Zoo (VeOmni — also FSDP + Ulysses-SP + EP, also omni-modal-targeted, also model-centric API) but ships with a much broader out-of-the-box example matrix: BAGEL with NSA, RAE-SigLip, SiT, WanVideo flow-matching, and dLLM are all first-class trainers in one repo, not just LLM/VLM. Where Scaling and Optimizing Frontier Model Training (Fireworks AI) productizes 4D parallelism (FSDP/PP/CP/EP) at the frontier (1T MoE, 1M context) and SpectraX — True MPMD Pipeline Parallelism for JAX takes the JAX-MPMD route, lmms-engine is the cheap-to-fork PyTorch reference for the same parallelism stack at the research-lab scale — closer in audience to KnapFormer: An Online Load Balancer for Efficient Diffusion Transformers Training and the Qwen3-VL recipes in Qwen3-VL — Multimodal LLM series (Dense + MoE, Instruct + Thinking). The Muon-by-default, Liger-by-default, packing-by-default stance is the same recipe Luma researchers have been seeing emerge across QK-Clip: Taking Muon Further on the Scaleup Journey, microsoft/dion — distributed Muon, Dion2, Dion, and NorMuon orthonormal optimizers, and How to Make LLM Training Faster with Unsloth and NVIDIA — this is the integrated-into-one-engine version.

See also

Section titled “See also”