From Pixels to Words: Towards Native Vision-Language Primitives at Scale

TL;DR

NEO is a family of native (monolithic, encoder-free) vision-language models from SenseTime × NTU that takes pixels and words in through lightweight embedding layers — no ViT — and processes both through a single dense backbone built from “native VLM primitives.” With only 345M / 40M / 4M image-text pairs across pre-training / mid-training / SFT, NEO-2B and NEO-9B (built on Qwen3-1.7B/8B) approach modular VLMs like Qwen2.5-VL and InternVL3 on standard benchmarks, and substantially beat prior native VLMs (Mono-InternVL, HoVLE, EVE, Emu3, SAIL). The architectural payload is three pieces: (i) a Multi-Head Native Attention (MHNA) that expands Q/K head dimensions and uses bidirectional attention on visual tokens but causal on text; (ii) Native-RoPE with separate channels and base frequencies for T, H, W to keep spatial and temporal indexes decoupled; (iii) a pre-Buffer / post-LLM split used only during pre-training to scaffold visual learning, then merged into a unified backbone for mid-training and SFT.

Key claims

The traditional modular VE→Projector→LLM design carries strong inductive biases from pretrained visual encoders (resolution, aspect ratio, fine-grained sensitivity) that require extensive post-training to mitigate, motivating a from-scratch native alternative [§1, §2.1].
A native VLM primitive should jointly: align pixels and words in a shared semantic space, integrate the strengths formerly split across VE and LLM, and inherently support cross-modal encoding/alignment/reasoning in one module — operationalized as Flexible Position Encoding + MHNA + Native-RoPE [§1].
Multi-Head Native Attention expands Q/K head dimensions (~10% extra parameters) with decoupled T / H / W channels and QK-norm; K weights for H/W are zero-initialized so the model starts as a pure-text LLM and progressively activates spatial modeling [§3.1, Fig. 3(1)].
Native-RoPE assigns distinct base frequencies and channels to T, H, W (e.g., θ_T = 10⁶ vs θ_H = θ_W = 10⁴), eliminating the cross-axis interference and frequency mismatch that 1D-RoPE / M-RoPE suffer when T indexes span millions but H/W only hundreds [§3.1, Fig. 3(2), Eq. 2].
A pre-Buffer / post-LLM split — modality-shared pre-Buffer trained from scratch, post-LLM warm-started from the pretrained LLM — is used only during pre-training; the partition dissolves in mid-training and SFT, yielding a single monolithic backbone [§3.1, §3.2, Fig. 4].
Image tokens use bidirectional attention while text tokens stay causal, with the whole image treated as one autoregressive meta-unit; FlexAttention with CUDA-kernel modifications keeps variable-length block-wise attention efficient [§3.1, Fig. 3(c)].
Pre-training uses 345M web-scale image-text pairs (LAION-400M, COYO-700M, BLIP3o long captions, OpenImages, LAION-COCO + Wukong OCR) with a 3:7 language-to-multimodal ratio, training only PEL + pre-Buffer + new Q/K head dimensions to protect LLM language ability [§3.2 Pre-Training].
NEO-2B (Qwen3-1.7B base) reaches MMMU 48.6 / MMB 76.0 / SEED-I 74.2 / POPE 87.5 — matching InternVL3-2B (48.6 MMMU / 81.1 MMB) on knowledge tasks and beating it on perception, despite using <1% of InternVL3’s pretraining data [Table 1].
NEO-9B (Qwen3-8B base) reaches MMMU 54.6 / MMB 82.1 / MMStar 62.4 / SEED-I 76.3, outperforming every prior native VLM (best previous: SAIL at MMB 70.1) and approaching Qwen2.5-VL-7B (MMB 83.5) [Table 1].
On document/OCR/chart benchmarks NEO is competitive but not best: NEO-9B reaches AI2D 83.1 / DocVQA 88.6 / ChartQA 82.1 / OCRBench 77.7 — close to but behind InternVL3 (DocVQA 92.7) and Qwen2.5-VL (DocVQA 95.7), and NEO-9B underperforms NEO-2B on DocVQA / InfoVQA, which the authors attribute to corpus limitations rather than the architecture [Table 2, §4.2].
Pre-Buffer depth saturates beyond ~8 layers; 12 layers is selected for the 2.2B model and 6 for the 9B to roughly match public vision-encoder capacity while keeping accuracy on the curve [§4.3, Fig. 5].

Method

NEO replaces the standard ViT-MLP-LLM stack with a single dense transformer assembled from “native VLM primitives.” Images enter through a lightweight Patch Embedding Layer (two Conv layers with stride 16 then 2, plus GELU, plus 2D sinusoidal positional encoding) that produces 32×32-pixel visual tokens; text enters through the LLM’s own WEL. Visual and textual tokens are concatenated and propagated through the same backbone — primitives that wrap the original LLM’s RMSNorm + SwiGLU with two pieces of new machinery. MHNA decouples temporal (T), height (H), and width (W) channels across Q and K, expanding the head dimension by ~10% and zero-initializing the H/W K weights so that pretrained linguistic behavior is preserved at initialization and spatial modeling is learned through training. Native-RoPE assigns the original LLM’s high base frequency to T (covering millions-long sequences) and a separate, lower base frequency to H and W (covering hundreds of pixels), with channel allocation that prevents zeroing H/W indexes from corrupting text-only attention. Image tokens get bidirectional masking inside the block while text stays causal, and FlexAttention with custom CUDA kernels handles the variable-length blockwise pattern.

The backbone is split during pre-training into a modality-shared pre-Buffer (12 native primitives for NEO-2B, 6 for NEO-9B) and a post-LLM warm-started from Qwen3-1.7B / 8B. Pre-training only updates the PEL, the pre-Buffer, and the new Q/K dimensions in the post-LLM — protecting the pretrained language weights. Mid-training and SFT then unfreeze the entire model and dissolve the pre-Buffer / post-LLM distinction, letting the unified backbone autonomously allocate capacity. Training uses three stages: 345M pretraining samples, 40M mid-training samples from InternVL-1.5’s corpus (captioning + conversation + detection + OCR), then 4M bilingual instructions for SFT — all with next-token-prediction loss, all end-to-end after pre-training.

Results

NEO-2B (Qwen3-1.7B base, 345M / 40M / 4M data): MMMU 48.6, MMB 76.0, MMVet 49.6, MMStar 54.2, SEED-I 74.2, POPE 87.5, HallB 43.1, AI2D 80.1, DocVQA 89.9, ChartQA 81.2, OCRBench 77.1 — matches or exceeds Qwen2.5-VL-2B and InternVL3-2B on MMMU / SEED-I / POPE / AI2D despite ~1/20th the pretraining data [Tables 1, 2].
NEO-9B (Qwen3-8B base, same data budget): MMMU 54.6, MMB 82.1, MMStar 62.4, SEED-I 76.3, AI2D 83.1, DocVQA 88.6 — the strongest native VLM on every benchmark in the comparison table, beating the next-best (SAIL-7B, EVEv2, BREEN) by 10+ points on MMVet and MMStar [Tables 1, 2].
vs the in-house “Encoder-Based” baseline (Qwen3 + InternViT-300M, same data): NEO-2B beats it on MMMU (48.6 vs 47.1), MMVet (49.6 vs 37.4), MMStar (54.2 vs 52.7), SEED-I (74.2 vs 73.6); NEO-9B trails it slightly on MMMU (54.6 vs 54.1) and DocVQA (88.6 vs 92.1) — encoder-free pays at small scale, parity or slight cost at 8B [Tables 1, 2].
Pre-Buffer depth ablation: average accuracy across 10 benchmarks rises monotonically up to ~8 layers, saturates beyond — chosen depth (12 for 2B, 6 for 9B) sits in the saturation region [§4.3, Fig. 5].
Limitations: NEO underperforms top modular VLMs on knowledge-heavy MMMU and OCR-heavy InfoVQA / TextVQA at the 8B scale, and NEO-9B falls below NEO-2B on DocVQA / InfoVQA, which the authors attribute to insufficient OCR-heavy data at the 9B training mix rather than the native architecture [§4.2].

Why it’s interesting

NEO is the architectural progenitor of NEO-unify: Building Native Multimodal Unified Models End to End — the wiki has the follow-up that adds a generation pathway via a Mixture-of-Transformer, but until now the foundational native-VLM paper was missing. Filing it closes that gap: every claim NEO-unify makes about “pixels and words in a shared semantic space without a VE” inherits from this paper’s MHNA + Native-RoPE + pre-Buffer recipe, and NEO-unify’s 31.56 PSNR reconstruction result is only meaningful once you know NEO already proved competitive understanding with the same primitives at 345M training pairs. It also directly substantiates Future of multimodal lies in native pixel-space unified architectures (Yuwei Niu quote-RT of NEO-Unify)‘s “native pixel-space unified architectures are the future” claim with concrete numbers. The comparison point that sharpens this is Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generation — Tuna-2 makes a similar pixel-embeddings-beat-VEs argument but for understanding+generation in one stack; NEO is the cleaner controlled experiment because it isolates the architectural choice (encoder-free monolithic VLM) from the multitask objective. The data-efficiency claim — modular-VLM-parity at ~1/20th the pretraining data — is also the strongest existing evidence that the inductive-bias overhead of carrying a pretrained ViT is real and measurable.