LingBot-VA 2.0: Native Video-Action Pretraining for Generalizable Robot Control

TL;DR

LingBot-VA 2.0 (Robbyant / Ant Group) is the second-generation causal video-action foundation model from the same lab as Causal World Modeling for Robot Control (LingBot-VA), and unlike v1 it is pretrained natively — the full stack (semantic visual-action tokenizer + causal DiT + action expert) is built from scratch for embodied control instead of adapting off-the-shelf video-generation components. Four design bets carry the paper: (i) a semantic visual-action tokenizer that replaces the pixel-reconstruction VAE with a ViT autoencoder aligned to a frozen Perception Encoder and paired with a self-supervised latent-action tokenizer; (ii) native causal pretraining on web-scale image/video (rather than retrofitting a bidirectional backbone into causal form, which is v1’s approach); (iii) a sparse MoE video stream (13B total / 1.9B active, 128 experts / top-8 with DeepSeek-V3-style auxfree balancing) inside a Mixture-of-Transformers architecture where the action expert stays dense; and (iv) Foresight Reasoning, an asynchronous inference scheme that predicts future latents in parallel with action execution and re-grounds each rollout against the latest real observation via learned forward dynamics — reportedly reaching 225 Hz peak closed-loop control on real hardware after few-step distillation + quantization. Positioned as the native-pretraining counter-recipe to π*0.6’s action-RL bet and Embodied-R1.5’s unified-VLM bet on the VLA Models page, and as a direct architectural evolution of v1’s causal video-action framing.

Key claims

Native from-scratch pretraining avoids the bidirectional-to-causal retrofit that v1 (LingBot-VA) required, keeping pretrained priors intact rather than eroding them via robot-data finetuning [§1, §2.1.2].
The semantic visual-action tokenizer aligns ViT-autoencoder latents to a frozen Perception Encoder via temporally-pooled MSE loss, and separately learns compact latent-action tokens by training an inverse-dynamics + forward-dynamics pair on unlabeled video with a bottleneck constraint, yielding paired (z, ℓ) tokens as the training substrate for the second stage [§2.2.1, §2.2.2, Fig. 2].
The MoE-13B-A1.9B video stream matches a Dense-5B baseline at step-aligned training loss (dense slightly lower) and matches it at wall-clock-aligned final loss — sparse capacity is added without an optimization penalty under the training budget [§2.3.2, Fig. 4].
Standard teacher-forced next-chunk supervision is myopic: adjacent latent chunks in high-fps video are visually similar enough that the loss rewards copying appearance rather than modeling dynamics. Multi-Chunk Prediction (MCP), imported from Next Forcing, supervises 3 future chunks at once via lightweight auxiliary heads at intermediate transformer layers [§2.3.3, Fig. 5].
Foresight Reasoning issues the next predicted-latent asynchronously and re-grounds each new prediction on the freshest real observation through a learned forward-dynamics step of the policy itself — a corrective mechanism naïve async schemes (which condition on stale imagined latents) lack; shares the concurrent-inference motivation of GEN-0’s Harmonic Reasoning but grounds it in a learned world model [§2.3.7].
Sparse MoE + few-step consistency distillation + Foresight Reasoning + quantized deployment combine to a reported 225 Hz peak asynchronous execution frequency on real robots for closed-loop control [§1, §2.4].
Real-world deployment claims few-shot generalization from only 10–15 demonstrations, cross-embodiment transfer without large-scale re-collection, and zero-shot execution of some new tasks — attributed to acquiring control knowledge at web-video scale rather than at robot-dataset scale [§1].
Reports outperforming π0.5 and LingBot-VA (v1) “by a substantial margin” across simulation and real-world evaluations [§1]; specific numbers are in the paper’s experiments section beyond the extracted range.
A hierarchical VLM planner (LoRA-fine-tuned VLM at ~2 Hz on an asynchronous shared buffer) decomposes long-horizon goals into a 4-field JSON schema (done / instruction / generation_instruction / local_scene_description); the low-level policy consumes the three textual fields, and boundary-crossing sampling produces the training data [§2.3.1].
In-context learning from human demonstration videos as visual task prompts is a first-class conditioning mode, with human–robot co-training bringing egocentric human videos into the shared world model under a multi-task recipe [§2.3.4, §2.3.5].

Method

The stack is two-stage. Stage 1 trains a semantic visual-action tokenizer: a ViT autoencoder with full spatial attention within each frame and causal attention across frames encodes the first frame as 16×16 patches and subsequent frames as 4×16×16 tubelets, with a symmetric decoder. Reconstruction is trained with pixel + perceptual + adversarial loss, and semantic alignment adds an MSE loss between temporally-pooled encoder latents and a frozen Perception Encoder’s features (via a learnable projection). On top of the frozen visual tokenizer, an inverse-dynamics model qφ(z_t, z_{t+1}) → ℓ_t produces a compact latent-action variable and a forward-dynamics model fψ(z_t, ℓ_t) = K_t z_t + δ_t decodes it into a transport-map-plus-residual predictor of the next latent; training uses forward prediction + backward-consistency losses on unlabeled video, with the bottleneck on ℓ_t preventing state copying (following RepWAM).

Stage 2 pretrains a causal video-action DiT on the resulting (z_{0:N}, ℓ_{0:N-1}) sequence. The Mixture-of-Transformers keeps two experts sharing one causal self-attention: the video expert has its dense FFN replaced by a routed MoE with N_e=128 and k=8 group-limited top-k routing plus a shared expert (Eq. 15–18), and uses the DeepSeek-V3 / Loss-Free Balancing / Moonlight-style auxfree correction bias update (Eq. 19) rather than an auxiliary balancing loss. The action expert stays dense. Both experts are flow-matching predictors: the video expert predicts the next visual latent given past latents and past actions; the action expert predicts the next action chunk a_t ≡ ℓ_t given past latents, past actions, and the predicted next latent — i.e. inverse dynamics over the predicted transition, in the parameterization inherited from v1’s MoT recipe (Eq. 6–8). Multi-Chunk Prediction adds three lightweight auxiliary heads that predict next¹/²/³ chunks from mid-network representations at layers L=20/12/4, so that the same causal representation captures trajectory-level dynamics rather than short-term appearance continuity.

At inference, Foresight Reasoning overlaps prediction with execution: while the robot is executing action chunk a_t, the model is already predicting z_{t+1} and a_{t+1} — but crucially each new prediction is re-grounded on the freshest real observation through a forward-dynamics update inside the policy itself, so the async pipeline doesn’t drift. Post-training distills both experts into few-step samplers and applies quantization for real-time control; the hierarchical VLM planner runs asynchronously at ~2 Hz on a separate buffer.

Results

Only architectural / training-loss / claim-level results are visible in the extracted range; per-task benchmark numbers live in the paper’s experiments section beyond the 30KB window. The concrete extracted results are:

MoE vs dense scaling: MoE-13B-A1.9B and Dense-5B follow near-identical training loss trajectories at step-aligned comparison (dense slightly lower) and match at wall-clock-aligned final loss (Fig. 4) — evidence the sparse video stream expands capacity without paying an optimization tax under the training budget.
Peak inference throughput: 225 Hz peak asynchronous execution frequency for closed-loop control on real robots, attributed to the combination of sparse MoE inference + few-step consistency distillation + Foresight Reasoning + quantized deployment [§1].
Data efficiency: claimed few-shot generalization from 10–15 demonstrations across complex manipulation tasks, cross-embodiment transfer without large-scale re-collection, and zero-shot execution in some settings [§1].
Baseline comparisons: outperforms π0.5 and LingBot-VA (v1) “by a substantial margin” across simulation and real-world evaluations [§1]; specific per-task tables not in the extracted range.

The v1 baseline for context: LingBot-VA (v1) reports 92.9 / 91.6 RoboTwin 2.0 Easy/Hard SR and 98.5 LIBERO average from the GitHub README (LingBot-VLA: A Pragmatic VLA Foundation Model-adjacent repo). v2 is claimed to beat v1 “by a substantial margin” — the concrete numbers are pending a fuller read.

Why it’s interesting

Three axes matter for the VLA Models and World Foundation Models pages:

The “native pretraining” recipe is a fifth structural bet on the VLA page. The concept page currently tracks (a) action-pretraining-at-scale (π*0.6), (b) clean teleop data (Spirit-v1.5), (c) unified-VLM pointing (Embodied-R1.5), and (d) frozen-WFM + small action expert (µ₀ / ImageWAM / World Pilot). This paper adds a fifth: keep the WFM-inside-VLA unification of v1, but pretrain the whole stack natively for causality and control from scratch rather than adapting a bidirectional video generator. Directly counter-positioned to LingBot-VLA 2.0 (From Foundation to Application: Improving VLA Models in Practice (LingBot-VLA 2.0)) — the same lab’s other July 2026 paper takes the pragmatic π-style route (scale robot data + auxiliary future-frame proxy loss on a Qwen-VL backbone), while this paper doubles down on the architectural-unification bet. The Robbyant lab is running both experiments in parallel.
Concrete answer to two open questions on the WFM page. Sitzmann’s action-conditional-WFM chicken-and-egg question is addressed here by bootstrapping the action signal from unlabeled video itself via the latent-action tokenizer’s inverse-dynamics objective — no paired action-observation data required for the world-model half. And the Foresight Reasoning + re-grounding recipe is a concrete answer to the “how does an autoregressive WFM stay coupled to reality under async execution” open question that Next Forcing: Causal World Modeling with Multi-Chunk Prediction and Causal World Modeling for Robot Control (LingBot-VA) both raise but neither fully solves.
First filed MoE-in-video-action-model instance. Existing MoE work in the wiki is either LLM-side (LongCat-Flash Technical Report, DeepSeek-V3) or T2I-side (NeuROK: Generative 4D Neural Object Kinematics-adjacent); this is the first filed paper putting a DeepSeek-V3-style routed MoE (128 experts / top-8 / auxfree balancing) inside a video-generation stream used as a policy backbone, and it reports parity with a matched dense baseline at both step-aligned and wall-clock-aligned loss.

The paper is a natural sequel-anchor between Causal World Modeling for Robot Control (LingBot-VA) (v1, the “adapt an existing video generator” position) and the still-open question of whether native pretraining actually beats retrofit at real-robot scale.