Emu3.5: Native Multimodal Models are World Learners

TL;DR

Emu3.5 is BAAI’s 34B decoder-only transformer trained end-to-end with a single next-token-prediction loss over ~13T interleaved vision-language tokens, ~55% of which come from sequential keyframes + ASR transcripts of ~63M internet videos (~790 years of footage). The model treats vision and language as one token stream — no modality adapters, no task heads — and after SFT + large-scale RL it claims parity with Gemini 2.5 Flash Image (Nano Banana) on T2I/X2I and superiority on interleaved generation, visual narrative, visual guidance, and world exploration. The headline inference contribution is Discrete Diffusion Adaptation (DiDA), which converts the AR visual-decoding path into bidirectional parallel prediction for a claimed ~20× per-image speedup at matched quality. Weights, code, and tokenizer are open on HuggingFace.

Key claims

A 34.1B decoder-only transformer (64 layers, hidden 5120, GQA 64/8, QK-Norm, SwiGLU, RoPE) trained from a Qwen3 initialization on ~13T multimodal tokens with a single next-token-prediction objective achieves the paper’s headline capabilities — no modality-specific heads, single vocabulary of 282,926 tokens (151,854 text + 131,072 vision) [§2.2, Table 1].
~55% of the pre-training mix is video-interleaved data: keyframes from PySceneDetect (1 frame per ≤5s scene or every 5s for longer scenes, ~0.27 keyframes/sec on average) interleaved with Whisper-large-v2 ASR transcripts, timestamp-aligned — ~63M videos / ~790 years of footage across education, How-to, science, gaming, travel, animation [§3.1.1].
A 455M-parameter IBQ visual tokenizer with codebook size 131,072 and 8× downsampling, distilled from SigLIP via REPA, uses one-fourth the tokens per image of Emu3 at equal or better reconstruction; an optional diffusion-based image decoder doubles output resolution and is LoRA-distilled from many denoising steps down to “a few” for ~10× decode speedup [§2.3].
DiDA (Discrete Diffusion Adaptation) converts the AR visual-decode path into bidirectional parallel prediction over visual tokens while keeping text decoding sequential, delivering ~20× per-image inference speedup without sacrificing quality, using only “a few billion tokens” of SFT + self-distillation data to adapt the NTP checkpoint [Abstract, §2.1].
Two-stage pre-training (10T tokens at 512² max, then 3T tokens with dynamic 1024-4096 visual tokens up to 1024² longest side), followed by SFT on 150B samples to establish the unified generation interface, followed by large-scale RL with multimodal rewards for long-horizon generation [§3.2, Table 2; §4.1].
Both training loss and 9 held-out validation losses (in-domain T2I/I2T/video-interleaved plus OOD ISG-Bench, OpenING, MMIE, plus three SFT-task validation splits) decline monotonically across S1, with the OOD curves continuing to fall as pre-training compute scales — taken as evidence that long-horizon interleaved pretraining yields generalization beyond training domains [§3.2, Fig. 7].
On X2I (any-to-image) tasks, Emu3.5 matches Gemini 2.5 Flash Image on image generation/editing and surpasses it on text-rich image generation; on interleaved generation (Visual Narrative, Visual Guidance) and world-modeling tasks (world exploration, embodied manipulation) it outperforms Gemini 2.5 Flash Image as a single 32B model [Abstract, §1].
Visual loss is down-weighted to 0.5 vs 1.0 for text loss to prevent the long visual sequences from dominating optimization — a concrete training-objective choice for joint-modality NTP at this scale [§3.2].
Three claims are framed as “observations” rather than ablations: (1) OOD validation loss continues to decrease as pretraining compute scales, (2) unified RL post-training establishes a shared interface where X2I high-fidelity + editing capabilities transfer into visual narrative and visual guidance, and (3) the NTP checkpoint can be cheaply converted to a bidirectional predictor without quality loss [§1].
Open-weights release covers three checkpoints: Emu3.5 (general-purpose interleaved + T2I/X2I), Emu3.5-Image (T2I/X2I specialist for best single-image quality), and Emu3.5-VisionTokenizer; both LM checkpoints are pure NTP at filing time, with DiDA-accelerated weights flagged as a future release [GitHub README §1].

Method

The architecture is a single decoder-only transformer over a unified 282K-token vocabulary that mixes Qwen text tokens with 131K visual tokens emitted by a 455M-parameter IBQ tokenizer (8× downsampling, SigLIP-distilled via REPA). Training is plain next-token prediction with a 0.5/1.0 visual/text loss weighting. The training mix is dominated by long-horizon video-interleaved sequences: PySceneDetect keyframes timestamp-aligned with Whisper-large-v2 ASR transcripts from ~63M internet videos, augmented in Stage 2 with LLM-generated visual captions (Qwen2.5-VL-7B), semantic-segment summaries, and unified multimodal summaries.

Pre-training runs in two stages — 10T tokens at fixed 512² (S1), then 3T tokens with dynamic 1024–4096 visual tokens per image (S2) — using TP=8, CP=2, batch 448, sequence length 32,768 on the FlagScale framework. Post-training is two-stage as well: SFT over 150B samples spanning T2I, X2I, language, VLQA, visual narrative, visual guidance, world exploration, and embodied manipulation, followed by large-scale RL with multimodal rewards aimed at long-horizon generation. DiDA then adapts the NTP checkpoint into a hybrid decoder — sequential text, bidirectional parallel visual — using “a few billion tokens” of SFT + self-distillation data, yielding the headline ~20×-per-image inference speedup. An optional diffusion image decoder doubles output resolution and is LoRA-distilled to a small number of denoising steps; a separate DiT-based video decoder lifts generated keyframe tokens into continuous video conditioned on per-frame text segments.

Results

Matches Gemini 2.5 Flash Image (Nano Banana) on image generation and editing (X2I); surpasses it on text-rich image generation and on interleaved generation tasks (visual narrative, visual guidance) — as a single 32B unified model, against a closed-source frontier baseline [Abstract, §1].
DiDA delivers ~20× per-image inference speedup without sacrificing performance vs token-by-token decoding, claimed as the first autoregressive image model to rival closed-source diffusion models in both speed and quality [Abstract].
Optimization dynamics: train + 9-set validation loss curves monotonically decline through 700k steps × 32,768 sequence length × 448 batch in S1; OOD validation loss (ISG-Bench, OpenING, MMIE) continues falling as compute scales — taken as a generalization claim [§3.2, Fig. 7].
Tokenizer reconstruction uses one-fourth the tokens of Emu3 at the same image size, with the diffusion decoder option giving better local detail (text regions, faces) at 2× resolution [§2.3].
The released README defers quantitative leaderboard tables to the linked paper PDF; the GitHub repo currently ships the NTP-only weights and notes that each image takes several minutes to generate without DiDA [GitHub README §1].

Why it’s interesting

Emu3.5 is the most ambitious single-stream NTP-only unified multimodal model the wiki has filed at this scale: it bets that long-horizon video-interleaved data is the right substrate, that joint pretraining + RL transfers across modalities, and that DiDA recovers the latency lost to AR visual decoding. This is a sharp contrast with the Unified Multimodal Models cluster’s recent direction, where the dominant recipes either decouple MLLM + DiT (DuoGen: Towards General Purpose Interleaved Multimodal Generation), distill understanding into generation, or go encoder-free MoT (NEO-unify: Building Native Multimodal Unified Models End to End) — Emu3.5 instead doubles down on the single-stream-token-NTP path Janus / Show-o pioneered, scales the data axis by 100× via internet video, and uses DiDA to retrofit parallel visual decoding rather than designing for it. It complements InfinityStar: Unified Spacetime AutoRegressive Modeling for Visual Generation‘s spacetime-AR direction by using interleaved frames + transcripts rather than spacetime-token rasterization, and complements Lumina-DiMOO: An Omni Diffusion Large Language Model for Multi-Modal Generation and Understanding by arriving at parallel visual decoding through post-hoc adaptation of an NTP checkpoint rather than from-scratch masked diffusion. The world-model framing also lands it in World Foundation Models as the first filed open-weights NTP-style WFM whose pre-training substrate is internet video transcripts — the closest open counterpart to the closed-flagship interactive-rollout WFMs (Genie 3, Runway GWM-1).