Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes

TL;DR

A Meta FAIR + Oxford controlled study of from-scratch unified multimodal pretraining (language + visual understanding + visual generation) that pins down four “physics-of-training” findings and validates them at 13.5B-MoE / 2T-token scale. Knowledge flow is asymmetric — language boosts vision universally, understanding is a strong prior for generation, generation is nearly neutral back. Modality synergy vs. competition is governed by task complexity (simple ↔ synergy, complex ↔ competition) and by architectural sharing (share attention + norm, decouple FFNs). Early joint unification beats every sequential schedule and cures a vision-laziness pathology in which late-added vision branches never fully activate. And the compute-optimal data mix is sharply asymmetric — L70 / U25 / G5 dominates the standard balanced mix while spending 5× fewer tokens on generation.

Key claims

Knowledge flow is directional, not symmetric: scaling language lifts every visual understanding axis and every generation metric; scaling understanding strongly boosts generation but mildly trades off language; scaling generation is roughly neutral back [§1, “Part 1A Modality transfer on real-world data”].
The direction holds across representations: RAE (SigLIP-2), Raw Pixels, and CLIP+VAE (continuous flow-matching) and UniTok-token AR generation all show the same transfer pattern [§1, “Part 1A”].
Concept-level transfer is concept-dependent, not modality-uniform: on controlled CLEVR, structural concepts (relation, size, count) transfer zero-shot from understanding → generation but not back, while low-level concepts (color, shape) fail zero-shot in both directions [§1, “Part 1B Controlled CLEVR concept transfer”].
Generation does still help understanding — but indirectly, by planting a latent prior that sharply speeds up color/shape recovery during fine-tuning rather than by direct zero-shot transfer [§1, “Part 1B”].
Modality synergy vs. competition is largely predicted by data/task complexity: at a fixed 100B split budget, simple partner data (solid backgrounds, patterns, repeated-letter text) pushes the other modality below its unimodal baseline (synergy), while complex data (video, natural images) pushes it above (competition), and the effect is symmetric across both modalities [§2, “Part 2A Data and task complexity”].
Architecturally, competition lives in the FFNs and synergy in shared attention/norm: fully shared dense blocks suffer the worst competition and fully split blocks lose synergy; decoupling only the FFNs while sharing attention and FinalNorm is the best of the five component-sharing designs tested [§2, “Part 2B Architectural choices for promoting synergy”].
The FFN-decoupled / attention-shared recipe generalizes across visual designs — RAE, Raw Pixels, CLIP+VAE, and AR (UniTok) all preserve modality synergy; Raw Pixels (no semantic prior) gives the largest language gain (ΔPPL −0.266 vs. RAE −0.211) [§2, “Part 2C Generalization across vision encoder designs”].
Delaying visual unification is costly: extending pure-language warm-up from 0B to 800B tokens (fixed 1T budget) plateaus language quickly but steadily degrades every visual axis (General, OCR & Chart, Vision-Centric, all generation metrics) [§3, “Part 3A Early unification vs. late unification”].
Joint training dominates all six possible sequential modality orderings on nearly every metric, and a 12.5% replay buffer only mildly mitigates forgetting without recovering the joint-training synergy [§3, “Part 3B Sequential vs. joint training”].
Vision laziness — introduced as a named diagnostic: as the language warm-up grows, the vision-side FFN activation L2 during training, image-wrapper token embedding L2, per-element inference activation of img_ffn, and attention on image tokens all fade together, so a more pretrained language trunk commits progressively less compute to vision [§3, “Part 3C Why late alignment creates vision laziness”, Axes 1–4].
Compute-optimal data mix is sharply asymmetric: language wants ~70% of tokens, understanding peaks near 25%, and generation reaches near-peak quality at just 5% of the budget — the L70/U25/G5 mix wins the grid on every axis, and cutting generation to 5% (Fix Lan sweep) even improves both understanding and generation vs. an L50/U25/G25 split [§4, “Part 4A Optimizing data-mixing ratios”, Fix MM / Fix Lan / Next tables].
At-scale validation (13.5B MoE / 1.5B active / 2T tokens): the asymmetric L70/U25/G5 “Full” recipe beats the L50/U25/G25 “Balanced” recipe on every metric — PPL 11.67 vs 11.97, Acc 54.31 vs 52.86, Understanding-Avg 43.08 vs 41.42, DPG 0.689 vs 0.676, GenEval 0.482 vs 0.467 — despite spending 5× fewer generation tokens; MoE beats a 3.5B dense baseline across the board; and early unification beats a Late Fusion baseline (vision added after 60% of training) on every axis [§4, “Part 4B Validating findings at scale”, model comparison table].

Method

The controlled protocol pretrains unified language + visual-understanding + visual-generation models from scratch at fixed 100B–1T-token budgets, testing continuous visual representations (RAE with SigLIP-2, Raw Pixels, CLIP + VAE) and autoregressive discrete UniTok tokens. Knowledge flow is measured in two settings: real-world (add more of one modality’s data on top of a fixed base of the other, read off effects on the partner’s capability) and controlled CLEVR (parallel understanding and generation streams from scene graphs, with target concepts removed from one stream while kept intact in the other — the leave-one-concept-out protocol for zero-shot transfer, plus fine-tuning acceleration measurements when zero-shot transfer fails). Modality synergy is disentangled along two axes: data/task complexity (simple → complex synthetic-to-natural sweeps on both sides of a 100B split) and Transformer-block component sharing (five designs: fully shared dense, FFN-only decoupling, FFN+Attention, FFN+FinalNorm, fully split). Vision laziness is instrumented at four axes — training-time img_ffn L2, image-wrapper token embedding L2, inference-time per-element img_ffn activation, and inference-time attention mass on image tokens — with only the starting language checkpoint (0B → 800B) varied under a fixed 200B-token late-alignment continuation. The scaling recipe is derived from a three-axis grid search at 1T tokens (Fix MM, Fix Lan, Next) and validated at 13.5B-total / 1.5B-active MoE on 2T tokens against a Balanced-Recipe baseline, a 3.5B Dense baseline, and a Late-Fusion baseline (vision added after 60% of training).

Results

Every finding replicates at 2T-token / 13.5B-MoE scale. Compared with the L50/U25/G25 Balanced Recipe, the L70/U25/G5 Full recipe delivers PPL 11.67 vs 11.97, Language-Accuracy 54.31 vs 52.86, Understanding-Gen 53.63 vs 51.50, Knowledge 40.11 vs 38.90, OCR 27.23 vs 25.15, Vision-Centric 51.33 vs 50.14, Understanding-Avg 43.08 vs 41.42, DPG 0.689 vs 0.676, GenEval 0.482 vs 0.467, CLIP-Sim 0.312 vs 0.310, and DiffLoss 0.272 vs 0.261. The Dense (3.5B) baseline is uniformly worse than the MoE Balanced Recipe on every axis (PPL 12.14 vs 11.97, Understanding-Avg 40.49 vs 41.42, DPG 0.667 vs 0.676). The Late Fusion baseline (vision added at 60% of training) is uniformly worse than Full on every axis and generally worse than Balanced too. The knee of the mixing-ratio sweep is at L70/U25/G5, and cutting generation from 25% to 5% (the Fix Lan sweep at L50) improves GenEval from 0.219 to 0.216–0.219 and Understanding-Avg from 37.0 to 38.2, evidence that language + understanding tokens are the load-bearing source of generation quality once the model has a small generation-data toehold.

Why it’s interesting

Sits directly next to Beyond Language Modeling: An Exploration of Multimodal Pretraining — same lab, overlapping author set (Junlin Han, Shengbang Tong, David Fan, Mike Lewis), and same “from-scratch unified pretraining as controlled science” ethos — but pushes past that paper’s scaling-asymmetry / RAE-vs-VAE / MoE-granularity findings into a mechanism-focused frame with named phenomena (knowledge flow, modality synergy, vision laziness) and a concrete deployable recipe (L70/U25/G5 + FFN-decoupled + early unification). Concretely sharpens two open questions from Unified Multimodal Models: (i) the understanding→generation transfer direction that UniMRG (Generation Enhances Understanding in Unified Multimodal Models via Multi-Representation Generation) exploited post-hoc is now argued to be a general pretraining-time asymmetry, and (ii) the “how much data does generation actually need” question — most published UMMs allocate 25–50% of budget to generation — is answered at 2T-token scale with 5% sufficing. The FFN-decoupled / attention-shared architectural recipe converges with ERNIE 4.5’s modality-isolated-router-plus-shared-attention finding in MoE Routing Design but arrives from the opposite direction (component sharing rather than expert routing). Vision laziness names a specific failure mode of the widespread late-vision-alignment recipe (LLaVA-style projector-tuning post-hoc on a pretrained LLM) that the Luma stack and most open UMMs share; the four-axis diagnostic gives a way to test whether a given late-fused model is exhibiting it.