MAGI-2 Preview: Scaling Video Generation Models Efficiently

TL;DR

Sand.ai’s MAGI-2 Preview is a ~114B-total / ~6B-active unified audio-video-text single-stream transformer built around what the post calls Ultra-Fine-Grained MoE: routing is done independently in 12 low-dimensional (256-d) head subspaces, each with its own router and 256-expert pool, top-6 per head — 3,072 head-local expert units per layer, of which 72 are activated per token. The stated goal is to scale MoE video generation past the point where standard token-choice MoE breaks down under long video sequences: cross-node all-to-all is regularized to fixed head-slice shapes (Hierarchical Head Parallel), thousands of narrow experts get a fused routing → grouped-GEMM → merge kernel (MagiMoE), and Muon is extended to orthogonalize head×expert matrix batches (MagiMuon). The data section is unusually direct — argues the community has been trapped in a filtering-centric pipeline that trims complex motion, multi-shot, and subtitled data to fit small models, and reports that preserving multi-shot and subtitled samples with precise multimodal annotation yields emergent cross-shot identity consistency and dialogue-synced subtitle effects without dedicated modules.

Key claims

The base architecture is single-stream: text, video, and audio tokens enter one unified sequence processed by self-attention only, inheriting the design validated by daVinci-MagiHuman rather than Magi-1’s autoregressive-chunk backbone [§1, §2].
MAGI-2 Preview scales to ~114B total parameters while keeping activated parameters at ~6B per token via Ultra-Fine-Grained MoE [§2, §2.2].
The MoE backbone is 40 Transformer layers: 4 dense boundary layers + 36 sparse middle layers with Multi-Head MoE; sparse layers split the 3,072-dim hidden state into 12 heads × 256-dim, each head has its own router and 256-expert pool, top-6 per head, expert FFN 256→1,280→256 with fused SwiGLU [§2.2, table in §2.2].
Each sparse layer therefore contains 12 × 256 = 3,072 head-local expert units, of which 12 × 6 = 72 are activated per token — sparser and finer-grained than the LLM-side DeepSeek-V3 / Kimi K3 recipes at token-level top-k routing [§2.2].
Head Parallel regularizes cross-node MoE communication: instead of the standard token-choice MoE where every rank sends dynamic Top-K-dependent payloads over the network, MAGI-2 dispatches activation slices along the fixed head partition; per-rank send/receive shapes are statically derivable and buffers preallocatable [§2.1, §3.1].
Head routing is data-dependent but the irregularity is contained inside each head owner rather than exposed to cross-device communication, so long video sequences do not blow up inter-node all-to-all payloads [§2.1].
Hierarchical Head Parallel maps activation exchange to InfiniBand and expert-state sharding to NVLink — inter-node handles fixed-shape head slices, intra-node uses ZeRO/FSDP-style full sharding to materialize expert weights on demand and reshard after computation, so a GPU responsible for a given head does not need to keep the head’s entire expert bank resident [§3.1].
MagiMoE fuses the routing → compact expert-wise token layout → grouped expert FFN → routing-weighted merge path as a single execution path with FP32 routing scores + BF16 expert weights, fused up/gate/SwiGLU projections, and a memory-efficient forward/backward for long video sequences — the claim is that generic Grouped GEMM MoE kernels are dominated by sorting / kernel launches / intermediate materialization when each expert unit receives few tokens [§3.2].
Load balancing uses auxiliary-loss-free expert bias applied independently per routing head, with expert-load statistics aggregated asynchronously on a separate CUDA stream to avoid interfering with the main compute path [§3.2].
MagiMuon preserves the head × expert matrix-batch layout in the optimizer: router gates are batched by head, expert up/gate/down projections flatten head-and-expert into the batch dimension, Muon orthogonalization runs independently per matrix, and the matrix batch is distributed across ranks — rather than concatenating everything into one artificial large matrix [§3.3].
Not all parameters use Muon: adapters, mHC, attention sinks, and gating-related parameters keep Adam-style updates while the main matrix parameters use MagiMuon [§3.3].
The “data filtering trap”: continual filtering to fit small-model capability makes filtered training data the capability ceiling of any larger model trained on it [§4.1].
MAGI-2 explicitly shifts the data pipeline center from filtering to high-throughput production + precise multimodal annotation, preserving multi-shot data, complex motion, multi-person interactions, subtitles, on-screen text, and complex audio relationships [§4.2].
Emergent capabilities from preserving relationships in data, without dedicated modules: retaining multi-shot videos yields identity consistency across shots; retaining subtitled data yields samples where character dialogue and songs appear with corresponding subtitle effects [§4.3].
MAGI-2 Preview is positioned as an intermediate validation of the three-way co-design — architecture, systems, data — rather than a scaling-law result; scaling curves, controlled ablations, and capability-boundary analyses are deferred [§5].
References cite Kimi K3’s Per-Head Muon as concurrent work: Per-Head Muon splits attention Q/K/V momentum matrices along attention heads and orthogonalizes independently, MagiMuon targets head-local expert matrices in Multi-Head MoE — different structures, same underlying principle [§3.3].

Method

MAGI-2 is a single-stream transformer that packs text, noisy video latents, and noisy audio latents into one token sequence and processes them entirely via self-attention, inheriting the design validated by Speed by Simplicity: A Single-Stream Architecture for Fast Audio-Video Generative Foundation Model (daVinci-MagiHuman) (daVinci-MagiHuman). The 40-layer backbone keeps 4 dense boundary layers top and bottom; the 36 middle layers are the Multi-Head MoE core.

Each sparse layer works as follows. A split projection maps the 3,072-dim hidden state into a “routed representation” partitioned into 12 latent heads of 256 dimensions each. Each head has an independent router and its own pool of 256 experts; each router selects Top-6 experts for its 256-d sub-token. Expert FFNs are narrow (256 → 1,280 → 256 with fused SwiGLU). The 12 head outputs are concatenated and mapped back to the transformer hidden state via a merge projection.

The key architectural bet is that routing 12 independent 256-d sub-tokens through per-head expert pools is qualitatively different from routing one 3,072-d token to top-k of a large expert pool: different representation subspaces of the same token can form different expert combinations, so the fine-grained expert catalog can compose in ways a coarse-grained MoE cannot.

Three systems components make this trainable at 114B on long video sequences. Hierarchical Head Parallel dispatches activation slices along the fixed head dimension across InfiniBand (send/receive counts statically derivable, buffers preallocatable) and uses ZeRO/FSDP-style sharding of expert weights, gradients, and optimizer states over NVLink. Because the head owner materializes the current layer’s expert bank on demand and reshards after, no single GPU keeps a head’s entire expert bank resident. MagiMoE fuses the whole routing → token layout → grouped expert FFN → weighted merge path into one execution path with FP32 routing scores + BF16 expert weights, avoiding per-narrow-expert kernel launch and intermediate-tensor overhead that would otherwise dominate. Auxiliary-loss-free expert bias is applied per routing head, with load statistics aggregated asynchronously on a separate CUDA stream. MagiMuon treats router gates and expert projections as batches of small matrices indexed by head × expert; Muon’s orthogonalization runs independently per matrix, and the matrix batch is distributed across ranks to balance optimizer compute. Adam is retained for adapters, mHC, attention sinks, and gating parameters.

The data pipeline is reframed around a “filtering trap”: preserving useful complexity (multi-shot, subtitles, complex audio) and describing it accurately via scalable multimodal annotation, rather than filtering out what the current model can’t fit.

Results

114B total parameters / ~6B active per token at the Preview scale [§2, §5].
Sparse core: 36 of 40 layers use Multi-Head MoE; expert configuration is 12 heads × 256 experts, Top-6 per head, FFN 256→1,280→256 [§2.2].
Qualitative results only in this Preview post: reported emergent cross-shot identity consistency from retaining multi-shot data, and dialogue-synced on-screen subtitle effects from retaining subtitled data [§4.3]. No quantitative benchmarks (VBench, human-preference Elo, MOVA Arena) are reported.
No training-throughput / MFU numbers are reported for the MagiMoE + Hierarchical Head Parallel + MagiMuon stack — only the qualitative claim that the co-design is what makes 114B tractable on long video sequences [§5].
No scaling curves, controlled ablations, or capability-boundary analyses reported — explicitly deferred to future work [§5].

Why it’s interesting

This is the second single-stream open joint-audio-video foundation model on the wiki after Speed by Simplicity: A Single-Stream Architecture for Fast Audio-Video Generative Foundation Model (daVinci-MagiHuman) (daVinci-MagiHuman, 15B) and now easily the largest — a ~7.6× jump in total parameters. Together with MiniMax H3 — Open-Weights Omni-Reference Video-Audio Generation Model Card (MiniMax H3, 33B dense single-stream) it’s the third data point against the dual-stream + bidirectional-cross-attention consensus tracked in Joint audio-video generation, and the first to reach frontier scale by going sparse-MoE rather than dense. The Ultra-Fine-Grained MoE recipe (12 head-local expert pools per layer, 256 experts each, Top-6 per head) is markedly sparser and finer-grained than the DeepSeek-V3 / Kimi K3 LLM-side recipes tracked in MoE Routing Design: 3,072 narrow expert units per layer vs a few hundred wider ones, with routing happening in 12 independent 256-d subspaces rather than at the full token width. This makes MAGI-2 the highest-sparsity-ratio MoE video generation model filed so far, and the systems justification (Head Parallel regularizing all-to-all shapes, MagiMoE fusing narrow-expert kernels, MagiMuon respecting the head × expert matrix batch) reads as the direct co-design answer to why prior video MoEs stopped at coarser granularity. The data-side “filtering trap” argument is a distinct contribution: it names something the video community has done implicitly and links preserving relationship-rich data (multi-shot, subtitled) to emergent cross-shot identity and subtitle-synced dialogue capabilities — a hypothesis that overlaps the Synthetic Training Data thread but pushes in the opposite direction (preserve real complexity rather than synthesize simpler substitutes). The absence of any quantitative benchmark in this Preview means the architectural + systems + data claims are so far unfalsified by public head-to-heads against LTX-2, MOVA, SkyReels-V4, MiniMax H3, or FLUX 3.