Beyond Language Modeling: An Exploration of Multimodal Pretraining

TL;DR

A FAIR/Meta + NYU controlled study of from-scratch unified multimodal pretraining in the Transfusion framework (next-token prediction for text + flow-matching diffusion for vision in a single decoder-only model), avoiding the confound of starting from a pretrained LLM. Four headline findings: (i) Representation Autoencoders (RAE — frozen SigLIP-2 / DINOv2 latents) are the right unified visual representation for both understanding and generation, beating VAEs; (ii) text and vision data are minimally interfering and often synergistic, with non-domain data improving VQA and world-modeling more than scaling in-domain data; (iii) Mixture-of-Experts induces natural modality specialization and benefits from very high granularity (16+ active experts); (iv) IsoFLOP scaling laws reveal a scaling asymmetry — vision is significantly more data-hungry than language, and MoE bridges this by letting capacity grow without adding active compute.

Key claims

Across all visual encoder families, the choice of representation barely affects text perplexity vs a text-only baseline — multimodal pretraining does not meaningfully degrade language capability [§3, Fig. 4].
Semantic encoders (SigLIP 2, DINOv2, WebSSL-L) outperform VAE encoders (SD-VAE, FLUX.1) on both visual understanding (VQA) and generation (DPGBench, GenEval) — invalidating the assumption that VAEs are necessary for generation [§3, Fig. 4].
Raw-pixel input is competitive on VQA but underperforms semantic encoders on generation; the gap is narrow enough to flag as a promising direction at larger compute [§3].
Modality-specific FFNs (text vs vision) uniformly improve over shared FFNs at the same active-parameter count — lower text PPL, better VQA, better generation [§2.2, Fig. 3].
Adding video to text minimally affects (and slightly improves) DCLM perplexity; adding image-text (MetaCLIP) hurts text PPL — the degradation is driven by caption distribution shift from pretraining text, not by vision itself [§4.1, §4.2, Fig. 6].
Cosine distance between caption corpora and DCLM correlates directly with text-PPL degradation (MetaCLIP 0.196, SSTK 0.215, MetaCLIP-Recap 0.286) [§4.2, Table 1].
Decoupling captioning sources by objective — MetaCLIP for I→T, SSTK for T→I — captures the strengths of each across all metrics [§4.2, Fig. 7].
Multimodal pretraining improves VQA more than scaling in-domain VQA data: 20B VQA + 80B heterogeneous data beats a 100B VQA-only baseline, including out-of-domain unlabeled video [§4.3, Fig. 9].
Multimodal pretraining benefits VQA across visual representations after Cambrian-7M finetuning, with semantic encoders consistently outperforming VAE encoders [§4.3, Fig. 10].
World-modeling capabilities (Navigation World Model setting: ATE, RPE) emerge primarily from general multimodal pretraining, not from NWM-specific data — pure video gives the largest gain among data-mix additions [§5.2, Fig. 13].
The model reaches competitive NWM performance with as little as 1% in-domain data when total budget is fixed at 200B tokens; performance saturates rapidly with NWM-data ratio [§5.2, Fig. 13].
Actions encoded as plain text numerical strings (no specialized continuous-vector adapter) suffice to integrate navigation into the unified model — and zero-shot OOD natural-language actions like “get out of the shadow!” produce coherent rollouts [§5, §5.3, Fig. 14].
MoE expert granularity matters: increasing granularity G from 1 to 16 (i.e. many small experts with Top-k routing) substantially improves both language PPL and vision diffusion loss at matched active compute; gains plateau beyond G=16 [§6.1, Fig. 15].
The optimal diffusion prediction target depends on the visual representation: x-pred outperforms u-pred for RAE (SigLIP 2) at all granularities; for VAE (FLUX.1), x-pred destabilizes text PPL at high granularity while u-pred scales cleanly [§6.1, Fig. 15].
Sparsity helps consistently: with active compute fixed (16 experts × hidden 512), growing total experts from 32 to 1008 (active ratio 50% → 1.6%) improves both modalities monotonically [§6.1, Fig. 16].
IsoFLOP analysis reveals a vision-vs-language scaling asymmetry: at matched FLOPs, vision is significantly more data-hungry per parameter than language; MoE narrows this gap by providing the model capacity language wants while accommodating vision’s data appetite [§7, Abstract].

Method

The system is a decoder-only Transformer trained from scratch on a mixture of text-only (DCLM), raw video (YouTube + curated 1 FPS), image–text pairs (MetaCLIP, Shutterstock), and action-conditioned navigation video (NWM, Bar et al. 2025). Text uses standard next-token cross-entropy with a LLaMA-3 BPE tokenizer; visual tokens are encoded frame-by-frame by a frozen visual encoder (default SigLIP 2 So400M, 224×224, others ablated) and trained with image-wise flow matching — sample t, interpolate x_t between x_0 and noise, predict velocity, with the noise schedule shifted toward noisier end following RAE (Zheng et al. 2026; Scaling Text-to-Image Diffusion Transformers with Representation Autoencoders). A single t is sampled per image/frame (Diffusion-Forcing-style independent noise). The architecture uses simple linear projections instead of U-Net adapters, modality-specific FFNs by default (later upgraded to MoE), and a FlexAttention hybrid mask — causal across language, block-bidirectional within each image/frame with causal attention between frames. Inference uses standard AR for text and 25-step Euler denoising for vision, decoded back to pixels by RAE / VAE-pretrained decoders, with CFG=3.0 (10% conditioning dropout during training).

Two evaluation directions: (a) unified pretraining ablations — visual representation, data mix, MoE design — measured on DCLM/Notes PPL, CC12M diffusion loss, DPGBench + GenEval generation, and 16-task Cambrian VQA after one Cambrian-7M finetune epoch; (b) NWM-style zero-shot planning using Cross-Entropy Method over 8-step horizons against an LPIPS goal metric, reporting absolute trajectory error and relative pose error.

Results

Default model is 2.3B total / 1.5B active (modality-specific FFNs); MoE configurations push to 13.5B total / 1.5B active. Pretraining budgets reach 1T tokens (520B text + 520B multimodal) for the data-composition study and 200B for NWM ablations. The headline qualitative findings: SigLIP 2 RAE beats FLUX.1 VAE on both VQA and GenEval; multimodal mixtures consistently beat text-only on VQA after finetuning and consistently beat domain-only on NWM; expert granularity G=16 is the knee; sparsity at fixed active compute improves both modalities monotonically; the IsoFLOP-derived vision/language scaling slopes differ in their data-vs-parameter exponents, with MoE bridging the gap. Quantitative tables and curves are in §3–7 of the paper; the asymmetric scaling-law fit (vision more data-hungry) is the central scaling result.

Why it’s interesting

This is the cleanest published treatment of the from-scratch unified-multimodal regime so far — Luma’s stack mostly inherits a pretrained LLM, which is exactly the confound this paper removes. Three things stand out. First, it pins down RAE as the right unified target at non-trivial scale: the Scaling Text-to-Image Diffusion Transformers with Representation Autoencoders T2I result (RAE beats VAE under MetaQuery) now has a companion result in the fully-unified Transfusion regime with text + video + actions. Second, it makes a direct empirical case that non-domain data is the dominant lever for both VQA and world-modeling — the 1% in-domain NWM result is the strongest filed datapoint that latent-predictive world-model capabilities are inherited from general video pretraining rather than learned from in-domain trajectories, complementing the predictive-WFM-as-reward thread in World Foundation Models (VJEPA-2 / VL-JEPA). Third, the MoE granularity / sparsity / scaling-asymmetry findings give the Hyperparameter scaling laws cluster its first joint vision+language IsoFLOP datapoint and contrast with the discrete-token MDM exponents in The Design Space of Tri-Modal Masked Diffusion Models (where larger MDMs were more data-efficient per parameter — here vision is uniformly more data-hungry than language).