LingBot-Video: Scaling Mixture-of-Experts Video Pretraining for Embodied Intelligence

TL;DR

LingBot-Video is the fourth paper in the Robbyant (Ant Group) LingBot family — a DiT-based video pretraining paradigm specifically tailored for embodied intelligence, positioned as “the first large-scale open-source MoE video foundation model” for robotics. It combines three integrated design choices: a single-stream diffusion transformer with sparse MoE FFN (DeepSeekMoE-style fine-grained + shared expert routing, auxiliary-loss-free load balancing), a robot-augmented data engine that mixes internet video with manipulation / navigation / egocentric robot footage, and a multi-dimensional reward system that adds physical-rationality and task-completion signals on top of standard aesthetic / prompt-following / motion objectives. Scaling experiments cover 1.3B dense through MoE 13B-A1.4B → 30B-A3B → 60B-A6B → 120B-A11B, with MoE 30B-A3B closely approaching Dense 14B at similar active FLOPs, and the MoE inference-latency gap over dense grows from parity at 16K tokens to 3.18× faster than Dense-30B at 1M tokens.

Key claims

Existing video foundation models optimize perceptual quality (realism, aesthetics, text alignment) but not physical correctness, controllability, or long-horizon state consistency — a gap that becomes the dominant failure mode when the video model is used as a world simulator for embodied AI [§1].
Three limitations of current video-for-embodied approaches — dense architecture, internet-only training corpora, aesthetic-only alignment objectives — are addressed jointly rather than one at a time [§1].
The base generator is a single-stream DiT (all visual + condition tokens share the same transformer blocks after per-modality projection), argued against dual-stream on Model-FLOPs-Utilization grounds: single-stream forms larger unified GEMM computations, avoids per-block concat/split of condition and latent tokens, and simplifies communication under sequence-parallel distributed training [§2.1].
Unified Input Formulation lets T2I, T2V, and TI2V share a single framework by treating image targets as single-frame (T=1) videos and using 3D MM-RoPE with non-overlapping temporal coordinate ranges for condition tokens (temporal-only (i, 0, 0)) vs. visual tokens ((L+1+f, h, w)) [§2.1].
QK-Norm (per-head RMSNorm on queries and keys before attention) plus adaLN-single modulation (one shared timestep MLP with per-layer trainable modulation tables) are used for scale stability [§2.1].
The FFN residual branch is replaced with a token-choice sparse MoE following DeepSeekMoE’s fine-grained-expert + shared-expert-isolation design; each expert is a SwiGLU MLP with intermediate width smaller than a dense FFN counterpart, and the shared experts act as an always-on common pathway [§2.2, Eq. 1].
Routing uses a sigmoid router with group-limited routing (128 routed experts split into groups; top-K groups scored by top-2 bias-corrected affinity; top-Kr experts selected within them) plus auxiliary-loss-free load balancing via a dynamic per-expert bias updated by the sign of load deviation [§2.2, Eqs. 2–5].
A sequence-wise auxiliary balance loss (Lseq, Eq. 7–10) is added on top of the online bias-correction because batch-level statistics can hide routing imbalance within individual long-sequence videos — critical for spatio-temporal token sequences reaching 1M+ tokens per video [§2.2].
Expert-count scaling (fixed 1.4B active, E∈{64, 128, 256}): validation loss improves with expert count, but the E=128→256 gain is marginal compared to E=64→128; E=128 is chosen as default [§2.3, Fig. 3].
Fine-grained vs coarse routing (fixed 13B total): MoE 13B-A1.4B-E128 (top-8 of 128) consistently beats MoE 13B-A1.5B-E64 (top-4 of 64) despite the coarse variant having higher active FLOPs, attributed to the larger combinatorial routing space and reduced knowledge-hybridity gradient conflicts [§2.3, Fig. 4].
Active-parameter-comparable scaling: MoE 13B-A1.4B substantially outperforms Dense 1.3B in both training and validation loss throughout training under a matched active compute budget [§2.4, Fig. 5].
Cross-compute dominance: MoE 13B-A1.4B and MoE 30B-A3B each outperform dense models with roughly twice their active parameter scale; MoE 30B-A3B closely approaches Dense 14B [§2.4, Fig. 6a].
Predictable scaling laws to 120B total: MoE 13B-A1.4B → 30B-A3B → 60B-A6B → 120B-A11B show consistent training-loss improvement with scale at aligned steps [§2.4, Fig. 6b].
Inference-efficiency scaling: MoE 30B-A3B latency vs. dense baselines at 1M tokens — 0.97× vs. Dense-3B (near-parity active-param equivalent), 1.50× vs. Dense-6B, 2.59× vs. Dense-14B, 3.18× vs. Dense-30B [§2.4, Fig. 7].
Text/multimodal conditioning is extracted by Qwen3-VL-4B and visual latents by the Wan2.1-VAE; the base generator is paired with a downstream cascaded refiner (not detailed in the extracted body) [§2.1].
Positioned as the first large-scale open-source MoE video foundation model targeting embodied intelligence, with website, GitHub, and Hugging Face checkpoints [§Abstract, §1].

Method

LingBot-Video’s base generator is a task-unified single-stream Diffusion Transformer. Visual patches and multimodal condition features (from Qwen3-VL-4B) are projected to a shared hidden dimension and concatenated along the sequence, then processed by stacked transformer blocks where all tokens share the same parameters. T2I, T2V, and TI2V are handled by treating images as single-frame video and using 3D MM-RoPE with non-overlapping temporal coordinates for condition vs. visual tokens. Each transformer block has QK-Norm on attention and an FFN residual branch that has been replaced with a sparse MoE layer: a fixed set of shared experts (always active) plus top-Kr routed experts selected from a pool of Nr routed experts via a sigmoid router, group-limited routing, and an auxiliary-loss-free dynamic bias for load balancing. A sequence-wise auxiliary balance loss is added because per-video routing imbalance can hide inside batch-level statistics. Timestep conditioning uses the adaLN-single design (one shared MLP + per-layer trainable modulation table).

Systematic MoE-recipe exploration is run along two axes under matched budgets. Under fixed active parameters (1.4B/token) the expert pool is scaled E∈{64, 128, 256}: validation loss improves monotonically but with diminishing returns, and E=128 becomes the default. Under fixed total parameters (13B) the granularity is compared — top-8 of 128 fine experts (A1.4B) versus top-4 of 64 coarse experts (A1.5B) — and the fine-grained variant consistently wins despite lower active FLOPs, attributed to the larger combinatorial routing space and reduced parameter-level gradient conflicts / knowledge hybridity. Scaling experiments then run four MoE sizes from 13B-A1.4B up to 120B-A11B and dense baselines at 1.3B / 3B / 6B / 14B / 30B, with training-loss curves indicating predictable scale-dependent improvement. On the inference side, MoE 30B-A3B is benchmarked against depth-matched dense variants at sequence lengths 16K → 1M: at 1M tokens it matches Dense-3B and progressively outpaces larger dense models, hitting 3.18× over Dense-30B.

Beyond the architectural body extracted here, the paper also describes (i) a data profiling engine that unifies internet-scale video with manipulation, navigation, and egocentric robot data, and (ii) a multi-dimensional reward system that extends aesthetic + prompt-following + motion criteria with physical-rationality and task-completion signals — both flagged as core contributions in the abstract though not covered in the extracted architecture-first sections.

Results

Headline scaling and efficiency numbers, all from the model-vs-baseline plots in §2.4:

Active-parameter-matched: MoE 13B-A1.4B decisively beats Dense 1.3B in both training and validation loss throughout training [Fig. 5].
Cross-compute dominance: MoE 13B-A1.4B and MoE 30B-A3B each beat dense models with roughly twice their active parameter scale; MoE 30B-A3B ≈ Dense 14B [Fig. 6a].
Predictable MoE scaling to 120B: MoE 13B-A1.4B → 30B-A3B → 60B-A6B → 120B-A11B (active params 1.4B → 3B → 6B → 11B) yields monotonically lower training loss at aligned steps [Fig. 6b].
Inference speed ratio at 1M tokens (MoE 30B-A3B vs. dense): 0.97× vs. Dense-3B, 1.50× vs. Dense-6B, 2.59× vs. Dense-14B, 3.18× vs. Dense-30B [Fig. 7].
Recipe ablations: E=128 (13B total, A1.4B) captures most of the E=64→256 capacity benefit; fine-grained top-8-of-128 beats coarse top-4-of-64 under matched 13B total budget [Figs. 3–4].
Full downstream evaluation (T2I, T2V, TI2V samples, physical-rationality reward comparisons, robot-oriented task completion) is signaled in the abstract and Fig. 1 but details lie beyond the extracted architecture-focused sections.

Why it’s interesting

LingBot-Video is the fourth and heaviest paper in the Robbyant / Ant Group LingBot family — completing the quartet with LingBot-VLA: A Pragmatic VLA Foundation Model (π-style VLA on Qwen2.5-VL-3B), Advancing Open-source World Models (LingBot-World) (LingBot-World — action-conditioned WFM on Wan2.2 MoE), and Causal World Modeling for Robot Control (LingBot-VA) (LingBot-VA — unified causal AR video+action) — and positions the group as running a coordinated three-scale stack: LingBot-Video is the video foundation model, LingBot-World the interactive world simulator built on it, LingBot-VA the policy that consumes it, and LingBot-VLA the end-to-end action head. Two things make it a distinctive datapoint for the wiki. First, it’s the first filed MoE video foundation model designed and scaled from scratch for embodied intelligence rather than aesthetics — the architectural recipe (DeepSeekMoE-style fine-grained + shared experts, auxiliary-loss-free bias + sequence-wise Lseq, group-limited sigmoid routing) directly imports LLM MoE practice into the DiT setting, and the reported cross-compute dominance and inference speedups suggest the MoE Routing Design cluster’s lessons transfer cleanly. Where existing MoE video work in the wiki either bolts MoE onto an existing dense checkpoint (LingBot-World’s 2-expert MoE initialized from Wan2.2) or focuses on inference-side systems (Mamoda2.5), LingBot-Video reports from-scratch scaling laws up to 120B/11B-active with fine-grained-vs-coarse and expert-count ablations at fixed active/total budgets — the DiT counterpart of the multi-axis MoE sweeps Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models and Margaret Li’s Slicing-and-Dicing thread are running on the LLM side. Second, the single-stream architecture choice contrasts sharply with the Dual-stream diffusion transformer cluster’s momentum — the paper’s argument is systems-first (MFU / GEMM shape / sequence-parallel communication), a different lens than the modality-decoupling arguments that motivated MM-DiT and SD3, and worth pinning as a design tension to watch as MoE scale in video pushes past 100B.