How Far Are We from Removing the Visual Encoder? Scaling Laws for Encoder-Free Multimodal Pretraining

TL;DR

A head-to-head scaling-law comparison of encoder-based and encoder-free multimodal LLMs (MLLMs) under matched compute, fitting Chinchilla-style loss–compute frontiers separately on the language and multimodal objectives. Three main findings: (1) removing the visual encoder shifts the multimodal compute-optimal allocation toward larger models while leaving the language allocation nearly unchanged; (2) the two architectures overlap on the text loss–compute frontier but diverge on the multimodal frontier — encoder-free underperforms at small scale yet is projected to catch up around $10^{22}$ FLOPs, within practical pretraining budgets; (3) without a vision encoder, the language model takes over the encoder’s role via vision-specific internal adaptation — bidirectional visual-token interactions become increasingly beneficial with compute, visual processing shifts to earlier layers, and expert routing for visual tokens becomes more concentrated. The advantage of a pretrained visual prior diminishes with scale, positioning encoder-free architectures as a serious pretraining direction.

Key claims

Removing the visual encoder shifts the compute-optimal allocation for the multimodal objective toward larger models, while the compute-optimal allocation for the text objective is nearly unchanged [Abstract, Finding 1].
The two architectures exhibit nearly overlapping loss–compute frontiers on the text objective, but diverge on the multimodal objective — encoder-free underperforms at small scale [Abstract, Finding 2].
Encoder-free MLLMs are projected to catch up to encoder-based MLLMs on the multimodal loss–compute frontier at approximately $10^{22}$ FLOPs, within practical pretraining budgets [Abstract, Finding 2].
Without a visual encoder, bidirectional interactions among visual tokens become increasingly beneficial as training compute grows [Abstract, Finding 3].
Without a visual encoder, visual processing shifts toward earlier layers of the language model as compute grows [Abstract, Finding 3].
Without a visual encoder, expert routing for visual tokens becomes more concentrated as compute grows — the language model develops specialized visual-processing experts [Abstract, Finding 3].
The advantage of the visual prior provided by a pretrained encoder diminishes with scale [Abstract, conclusion].

Method

The paper runs a controlled IsoFLOP + loss-vs-compute comparison of two architectures under matched training data and compute: an encoder-based MLLM (pretrained vision encoder + language model, in the standard MLLM configuration) and an encoder-free MLLM (raw pixels projected directly into the language model with no pretrained visual prior). Both are trained from scratch across model sizes and token budgets, and scaling laws are fit separately for the text objective and the multimodal objective — so the “how does removing the visual encoder change scaling” question decomposes into two comparable Chinchilla-style frontier fits per architecture. Finding (3) is then supported by architectural probes on the trained encoder-free models — attention-direction ablations across visual tokens, layer-depth attribution of visual processing, and MoE routing-concentration measurements for visual tokens as compute increases — to characterize how the language model internally compensates for the missing pretrained visual prior.

Results

Text loss–compute frontier: encoder-based and encoder-free are nearly overlapping — removing the encoder does not degrade the text scaling law [Finding 2]. Multimodal loss–compute frontier: encoder-free underperforms at small scale but the two frontiers converge, with the crossover projected at ~ $10^{22}$ FLOPs [Finding 2]. Compute-optimal allocation on the multimodal objective shifts toward larger parameter counts in the encoder-free setting [Finding 1]. Architectural probes on the trained encoder-free models show three consistent, compute-scaling trends: bidirectional visual-token interactions become more beneficial, visual processing migrates to earlier layers, and MoE expert routing concentrates for visual tokens [Finding 3].

Why it’s interesting

This is the direct scaling-law sequel to Scaling Native Multimodal Pre-Training From Scratch — where that paper decoupled the language and multimodal allocation laws inside a single encoder-free family and showed the multimodal law is composition-variant, this one compares encoder-based vs encoder-free head-to-head under the same decoupled-law framework and predicts the crossover point (~ $10^{22}$ FLOPs). Together they close a gap in the wiki’s unified-multimodal-models open question “does the encoder-free MoT approach scale beyond 2B preview”: there is now a compute-optimal fit that says yes and names the budget. The Finding-(3) architectural probes are the mechanistic counterpart to [[2608.05000-towards-physics-multimodal-pretraining-knowledge]]’s “vision laziness” pathology — that paper documented late-alignment vision-side FFN activation shrinking under a naive language-heavy warm-up, whereas this paper documents the opposite trend under encoder-free training: with compute, the language model grows dedicated visual pathways (earlier layers, concentrated experts, bidirectional attention). Sits next to From Pixels to Words: Towards Native Vision-Language Primitives at Scale (NEO, encoder-free VLM), Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generation (Tuna-2, pixel-native UMM), and NEO-unify: Building Native Multimodal Unified Models End to End (NEO-unify) as the fourth major encoder-free filing this cycle — but the first to give a when-does-it-pay answer in compute-optimal terms.