Emerging Properties in Unified Multimodal Pretraining

ListenBAGEL and the Case for Bottleneck-Free Multimodal Models10:32

Show notes

BAGEL is ByteDance's open-source unified multimodal model — a 7B-active, 14B-total Mixture-of-Transformer-Experts built on Qwen2.5, sharing self-attention between an understanding expert and a generation expert at every layer. We dig into the bottleneck-free architecture, the interleaved data recipe that dominates the training mix, and the paper's central claim: that scaling interleaved pretraining causes multimodal capabilities to *emerge* in stages. We also read the numbers the way a lab meeting would, and flag what's shown versus what's asserted. **Sources** - Emerging Properties in Unified Multimodal Pretraining (BAGEL)

Subscribe to the channel: https://podcast.lumalabs.link/27a25f51ea81d41a8253f46c6133a35e/feed.xml

TL;DR

BAGEL is ByteDance’s open-source foundational Unified Multimodal Model — a 7B-active / 14B-total Mixture-of-Transformer-Experts built on Qwen2.5, with one expert dedicated to understanding (SigLIP2 ViT input) and one to generation (FLUX VAE latents, rectified flow objective), sharing self-attention every layer for bottleneck-free interaction. It’s trained on ~7T tokens of interleaved text, image, video, and web data — including 45M temporally-grounded video interleaved sequences and 20M caption-scaffolded web interleaved documents — plus 500K reasoning-augmented examples for T2I, image manipulation, and conceptual edits. The paper’s central empirical finding is emergence: as interleaved pretraining scales, basic understanding + generation converge first, then free-form image manipulation and multi-view / world-navigation skills surface, and finally long-context reasoning starts to lift both understanding and generation together — capabilities absent from standard T2I benchmarks and revealed through the authors’ IntelligentBench.

Key claims

Bottleneck-free Integrated Transformer > External-Diffuser designs for UMMs at scale: shared self-attention between understanding and generation experts at every layer avoids the “compress LLM context into K latent tokens” information loss and is more amenable to scaling than adapter-mediated designs like [Show-o / MetaQuery / BLIP-3-o] [§2.1].
The MoT variant (both attention and FFN duplicated per modality; hard-routed by token type) consistently beats both dense and MoE (only FFN duplicated) at the 1.5B scale under matched FLOPs, with the gap most pronounced on generation MSE loss — evidence that understanding and generation objectives steer parameters toward distinct regions and benefit from decoupled capacity [§2.4, Fig. 3].
The generation expert uses rectified flow on FLUX-VAE latents; the understanding expert uses next-token-prediction on SigLIP2/NaViT tokens with 2D positional encoding; a shared QK-Norm stabilizes training [§2.2].
Generalized causal attention: per-image the model carries three token sets — noised VAE (for RF training, MSE loss target), clean VAE (KV-cached, seen by later generation/text), and ViT (unifies interleaved gen+und formats and empirically boosts interleaved-gen quality). Diffusion forcing adds independent noise levels across images with grouped full-attention inside consecutive-image groups [§2.3].
Data recipe is dominated by interleaved rather than paired data: 500M paired understanding, 1.6B paired generation, but the interleaved buckets (100M understanding, 45M video-derived, 20M web-derived) contribute the emergent capabilities [Table 1, §3].
Video-interleaved data is constructed by distilling a Qwen2.5-VL-7B captioner into a lightweight inter-frame captioner (30-token cap) and generating captions between sampled frame pairs — producing “temporally grounded” interleaved sequences that unlock world-navigation and 3D-manipulation behavior [§3.3.3, Fig. 4(a)].
Web-interleaved data uses a caption-first strategy: a Qwen2.5-VL-7B caption is inserted directly before each image as a conceptual scaffold, mitigating the loose alignment between web images and their surrounding text and improving image-generation controllability [§3.3.3, Fig. 4(b)].
Reasoning-augmented data (500K examples across T2I, free-form manipulation, and conceptual edits) is constructed by prompting VLMs with a DeepSeek-R1 reasoning-trace exemplar, so BAGEL learns to plan generations via language-based reasoning [§3.3.4].
Scaling with interleaved data yields a staged emergence pattern: basic understanding + high-fidelity generation converge first, then complex editing and free-form manipulation surface, then long-context reasoning benefits both — atomic skills synergize into compositional multimodal reasoning at scale [§1, Introduction].
BAGEL matches SD3 / FLUX.1-dev on T2I quality while beating top-tier open-source VLMs on standard multimodal understanding leaderboards, and extends qualitatively into world-modeling tasks (free-form manipulation, multiview synthesis, world navigation) that image-editing models cannot address [§1, Fig. 1].
CFG at inference drops text, ViT, and clean-VAE tokens independently with probabilities 0.1, 0.5, 0.1 respectively during training — supporting multi-axis classifier-free guidance in interleaved inference [§2.3].

Method

BAGEL is a decoder-only Mixture-of-Transformer-Experts backbone initialized from Qwen2.5. Text and ViT (SigLIP2) tokens flow through the understanding expert; VAE (FLUX) latents flow through the generation expert. Both experts share the self-attention operation at every layer (hard-routed by token type), so understanding and generation see each other’s KV without a bottleneck. Positional encoding is 2D for both ViT and VAE tokens; diffusion timestep is injected as an additive embedding on the initial VAE hidden state (no AdaLN). Text tokens use next-token-prediction (cross-entropy); VAE tokens use rectified flow (MSE on noised latents at loss-weight ratio 0.25 CE : 1 MSE).

The generalized causal attention interleaves three token sets per image — noised VAE (loss target, invisible to future tokens), clean VAE (KV-cached, visible to future tokens), and ViT tokens — with causal attention on text spans and bidirectional attention on vision spans. For multi-image generation, diffusion forcing gives each image its own noise level and groups consecutive images with full attention inside each group. Implemented via PyTorch FlexAttention for a 2× speedup over naive SDPA.

Data mixes across text (400M / 0.4T tokens), paired image-text understanding (500M / 0.5T), paired image-text generation (1.6B / 2.6T), interleaved understanding (100M / 0.5T), video-interleaved generation (45M / 0.7T), and web-interleaved generation (20M / 0.4T) — with the last three buckets being the distinguishing feature of the recipe. Reasoning-augmented data (500K examples in T2I, free-form manipulation, conceptual edits) is constructed with a VLM prompted by DeepSeek-R1 exemplars. Training runs four stages: alignment (5K steps), pre-training (200K steps, 2.5T seen tokens), continued training (100K steps, 2.6T tokens), and SFT (15K steps, 72.7B tokens).

Results

On standard T2I benchmarks BAGEL matches SD3 and FLUX.1-dev in generation quality while beating top open-source VLMs on multimodal-understanding leaderboards [§1].
Qualitatively, BAGEL extends beyond image editing into free-form visual manipulation, multi-view synthesis, and world navigation (referred to as world-modeling capabilities), evaluated in the paper’s proposed IntelligentBench [§1, Fig. 1].
The staged emergence pattern (basic → complex editing → long-context reasoning) is stated as a direct observation from scaling curves during pretraining, and is claimed to be more distinctly measurable on IntelligentBench than on GenEval-style benchmarks [§1].
MoT vs Dense vs MoE (1.5B controlled study, matched FLOPs): MoT achieves both fastest convergence and lowest final MSE on the generation objective; CE (understanding) is noisier but MoT still wins overall [§2.4, Fig. 3].

Why it’s interesting

This is the ur-paper for a whole line of derivative work already on the wiki: Hyper-Bagel: A Unified Acceleration Framework for Multimodal Understanding and Generation distills BAGEL’s diffusion branch and puts EAGLE-3 on its AR branch; LightBagel: A Light-weighted, Double Fusion Framework for Unified Multimodal Understanding and Generation proposes a lighter double-fusion alternative to the MoT design; and Towards Practical World Model-based Reinforcement Learning for Vision-Language-Action Models repurposes BAGEL as an off-the-shelf dynamics + reward model for VLA finetuning — extending Unified Multimodal Models into embodied RL. Shyamal’s Slack note also indicates BAGEL is the internal ByteDance lineage that would have been “Bagel 2” — meaning the model’s emerging-properties framing is deliberately downplayed in the paper title. The MoT-with-shared-attention pattern here also prefigures the SenseTime NEO-unify recipe (NEO-unify: Building Native Multimodal Unified Models End to End), which pushes the same principle to an encoder-free extreme (no ViT, no VAE, pixel I/O). And the 45M video-interleaved generation bucket is the earliest filed data recipe demonstrating that scaling temporally-grounded image-text interleaved sequences is what unlocks world-navigation and multi-view behavior in a UMM — a data-side complement to the architecture-side arguments on World Foundation Models.