Scaling Native Multimodal Pre-Training From Scratch

TL;DR

An IsoFLOP + training-curve-envelope study of native multimodal pretraining (decoder-only Transformer, MoE, single patch-embedding projection into continuous image tokens, no vision encoder) that fits Chinchilla-style compute-optimal scaling laws separately for the language and multimodal objectives. Headline finding: the two objectives obey different allocation laws. The language allocation is largely composition-invariant — text scaling exponents barely move as the multimodal token ratio $r_\mathrm{mm}$ grows. The multimodal allocation is strongly composition-variant — as $r_\mathrm{mm}$ increases, the optimal parameter exponent $a$ collapses and the token exponent $b$ rises, so dense multimodal data pushes the compute frontier toward more tokens and less parameters. A joint Pareto frontier over compute × $r_\mathrm{mm}$ gives concrete $(N^*, D_\mathrm{text}^*, D_\mathrm{mm}^*)$ configurations. Downstream: text-only performance is preserved across all $r_\mathrm{mm}$ , multimodal pretraining transfers positively to text-only spatial reasoning (widening with scale), and multimodal in-context learning emerges with model + data scale.

Key claims

Under a fixed compute budget $C \approx 6ND$ , minimum loss for each objective (language and multimodal) follows a power law in compute, and the compute-optimal model-size and token-count allocations follow $N^* \propto C^a$ and $D^* \propto C^b$ with objective-specific exponents $(a, b)$ [§2.1, Eq. 2].
Two independent estimators — IsoFLOP profiles (parabolic fit at fixed $C$ ) and training-curve envelopes (lower envelope across trajectories) — agree closely on the fitted exponents, indicating the exponents reflect the data distribution rather than a functional-form artifact [§2.2, Figs. 1–2].
The language allocation law is composition-invariant: IsoFLOP-derived exponents drift only slightly as $r_\mathrm{mm}$ increases, and the training-curve envelope shows no monotonic drift; parameter capacity needed to minimize the language loss is strictly determined by the effective language compute $C_L = (1-r_\mathrm{mm}) C$ [§3.1].
The multimodal allocation law is strongly composition-variant: as $r_\mathrm{mm}$ grows from small to large values, the optimal-model-size exponent for the multimodal objective drops substantially in both IsoFLOP and training-curve-envelope fits, flattening the parameter-scaling curve and shifting compute allocation toward data [§3.2].
Joint Pareto analysis over (compute × $r_\mathrm{mm}$ ) gives concrete deployable configurations: at low $r_\mathrm{mm}$ ( $\approx 0.05$ ) the optimal parameter allocation follows $N^* \propto C^{0.54}$ ; at high $r_\mathrm{mm}$ ( $\approx 0.7$ ) it collapses to $N^* \propto C^{0.42}$ with $D^* \propto C^{0.58}$ — larger unified multimodal models want proportionally more tokens rather than more parameters [§3.3].
Native multimodal pretraining preserves core language capabilities: across six model scales (71M–3B active) and $r_\mathrm{mm} \in [0, 0.23]$ , average accuracy on 16 text benchmarks deviates by less than one percentage point at every scale [§4.2, Fig. 8].
Cross-modal positive transfer to text-only spatial reasoning: on the strictly text-based SpatialEval subtasks (MazeNav, SpatialMap), multimodal-pretrained models beat the text-only baseline, and the gap widens with model scale — marginal at 71M / 128M, substantial by 3B [§4.2, Fig. 9].
Multimodal in-context learning emerges with scale: 3-shot vs 0-shot gain grows from ≈0 at 71M to +1.80 pts at A874M to +2.43 pts at A3B across 21 multimodal benchmarks; at 3B, accuracy monotonically improves with shot count [§4.3, Fig. 10].
In-context learning strengthens with training data: the shot-count hierarchy (3-shot > 1-shot > 0-shot) is absent early in training and only emerges late — and only for the A874M and A3B models; smaller models never open the shot-hierarchy gap even at end of training [§4.3, Fig. 11].
Few-shot benefits are task-specific: spatial-reasoning and structured-diagram benchmarks gain the most, while OCR- and recognition-oriented tasks plateau or degrade with in-context templates — consistent with the cross-modal spatial-transfer finding [§4.3, Fig. 12].
The framework decouples the training loss $L_{\text{total}} = (1-r_\mathrm{mm}) L_L + r_\mathrm{mm} L_M$ into two allocation problems and models the multimodal composition explicitly, which the prior single-aggregate-loss framing in [36] (Emu3.5-style analysis) does not — the divergence between the two allocation laws is invisible under an aggregate fit [§5].

Method

The base architecture is a decoder-only Transformer with auxiliary-loss-free MoE routing (23), trained from scratch on interleaved text + image data. Images are projected directly into continuous patch embeddings by a single embedding layer — no ViT/SigLIP/DINOv2 encoder, no VAE — with the same decoder-only trunk handling text tokens and image-patch embeddings jointly. Text loss is standard next-token cross-entropy on 250B web/book/paper tokens; multimodal loss is defined over 75B image-conditioned tokens drawn from web-crawled pairs and interleaved image-text documents. The scaling study sweeps six model scales (71M to 3B active parameters), multiple token budgets per scale, and multimodal ratios $r_\mathrm{mm} \in \{0, 0.05, 0.11, 0.18, 0.23\}$ (up to 75B/(250B+75B) ≈ 0.23 in the downstream evaluation; the scaling-law fits span a wider range).

Two independent estimators pin down the compute-optimal frontier: (a) IsoFLOP profiles — at each budget $C$ , fit a parabola in $\log N$ to the loss across models, extract the minimum $N^*$ , back out $D^* = C/(6N^*)$ , and regress on $C$ to get exponents; (b) training-curve envelope — pool loss-vs-compute trajectories across all scales, take the lower envelope, and regress $N^*$ and $D^*$ against $C$ . The compute frontier itself is fit with a log-sum-exp-parameterized power law $L(C) = L_\infty + (C/C_0)^{-\alpha}$ enforcing strict positivity. For the joint Pareto analysis, the paper pairs a composition-invariant language allocation law with a composition-variant multimodal allocation law — an asymmetric modeling choice justified by the empirical divergence, not imposed a priori. Downstream evaluation uses in-context templates (multiple-choice: lowest-perplexity option; open-ended: exact match; coding: Pass@1) with an image budget of 1536 tokens per image (512 in few-shot to fit templates and query into a 4K context).

Results

Scaling-law fits (IsoFLOP + training-curve envelope) agree on divergent language/multimodal allocation exponents; language exponents cluster near Chinchilla-style ( $a \approx 0.5$ ) across $r_\mathrm{mm}$ , while multimodal $a$ drops substantially at high $r_\mathrm{mm}$ (paper reports the projection $a \approx 0.42$ , $b \approx 0.58$ at $r_\mathrm{mm} = 0.7$ vs $a \approx 0.54$ , $b \approx 0.46$ at $r_\mathrm{mm} = 0.05$ ). Downstream: text-benchmark averages sit within 1 pt of the text-only baseline across all scales and all $r_\mathrm{mm}$ ; SpatialEval MazeNav + SpatialMap gap opens with scale, reaching several points by 3B; multimodal few-shot gain reaches +2.43 pts at 3B and continues trending upward with training tokens.

Why it’s interesting

This is the sharpest filed decomposition of multimodal scaling to date. Beyond Language Modeling: An Exploration of Multimodal Pretraining reported a scaling asymmetry — vision more data-hungry than language — via a single IsoFLOP fit on unified loss; this paper isolates the asymmetry by fitting language and multimodal exponents separately and shows the language law is essentially composition-invariant while the multimodal law is not. That’s a much stronger claim than “vision needs more data” — it says the language leg of a unified model scales like a pure LLM regardless of how much multimodal data is mixed in, so the entire compute-allocation question reduces to choosing $r_\mathrm{mm}$ and then running two decoupled Chinchilla-style fits. Sits directly next to Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes (same-lab-adjacent line, “physics of multimodal pretraining”) which arrived at an asymmetric data mix (L70/U25/G5) at 13.5B-MoE / 2T tokens — the two results are complementary: this paper says the scaling exponents diverge across modalities; the other says the token-share that maximizes downstream quality is heavily language-weighted at frontier scale. Also complements The Design Space of Tri-Modal Masked Diffusion Models (tri-modal MDM with $D^* \propto N^{0.476}$ , more data-hungry than AR) and Scaling Laws For Diffusion Transformers (DiT text-to-image, $N^* \propto C^{0.56}$ , slightly more parameter-heavy than data-heavy) — three from-scratch multimodal scaling regimes now fit end-to-end, with the exponents genuinely differing. Encoder-free design (single patch embedding, no ViT/VAE) puts this on the same architectural axis as NEO-unify (NEO-unify: Building Native Multimodal Unified Models End to End) and Inkling (Introducing Inkling — Thinking Machines' open-weights 975B/41B MoE multimodal reasoning model) — a growing cluster of frontier-scale evidence that the pretrained-vision-encoder era is optional for native UMMs.