Fuyu-8B: A Multimodal Architecture for AI Agents

TL;DR

Fuyu-8B (Adept, October 2023) is a vanilla decoder-only transformer based on Persimmon-8B, released open-weights under CC-BY-NC, that removes the vision encoder entirely: image patches are linearly projected directly into the first transformer layer, no ViT, no cross-attention, no adapter, no separate image-encoder pretraining stage. This is the earliest widely-known “encoder-free VLM” — the direct architectural ancestor of Tuna-2, NEO / NEO-unify, and the encoder-vs-encoder-free scaling-law study filed today. The design supports arbitrary image resolutions natively (image tokens in raster-scan order with a special newline sentinel, reusing text position embeddings), which is what leads to the ~30K tokens/image cost that Sid flagged in Slack — the missing pretrained visual prior is compensated for by very long visual token sequences fed directly through the LLM.

Key claims

• Fuyu is a vanilla decoder-only transformer with the same architecture as Persimmon-8B; there is no image encoder [Architecture].
• Image patches are linearly projected into the first transformer layer, bypassing the embedding lookup, so the LLM handles vision natively as extended token positions [Architecture].
• Fuyu supports arbitrary image resolutions by treating image tokens as an extension of the text token sequence, with a special image-newline character to mark line breaks and no separate image position embeddings [Architecture].
• Training requires no separate low- vs. high-resolution image phases and no separate image-encoder pretraining stage — the model is trained end-to-end [Training].
• Fuyu-8B improves over QWEN-VL and PaLM-E-12B on 2 of 3 shared benchmarks despite being smaller: VQAv2 74.2, OKVQA 60.6, COCO 141, AI2D 64.5 [Table 1].
• Fuyu-Medium (larger, not released) matches PaLM-E-562B on OKVQA (63.1 vs 66.1) with fewer than a tenth as many parameters [Table 1].
• The Fuyu-based internal models can perform bbox_to_text and text_to_bbox on UI screenshots, reading the text under arbitrary bounding boxes and localizing text on-screen for click targeting [§ UI capabilities].
• Standard VQA scoring (exact-match against a small answer list) systematically penalizes semantically correct answers — “snare” scored 0 vs. reference “drum”; “fish, carrots” scored 0 for a correct list [§ Question Answering Benchmarks].
• COCO CIDEr correlates poorly with internal evaluations — Fuyu-Medium is slightly worse than Fuyu-8B on CIDEr despite being a stronger model on the team’s internal eval [§ Captioning Benchmarks].

Method

The architecture is a subtractive one: start from the Persimmon-8B decoder-only transformer, delete the vision encoder, delete the adapter/connector, delete the cross-attention, and delete the image-specific positional embeddings. What remains is a linear projection from raw image patches into the transformer’s first-layer embedding space, in place of the token embedding lookup for image positions. Images are fed in raster-scan order (top-left to bottom-right, row by row) as extended sequences of visual “tokens”; a single learned image-newline sentinel separates rows so the model can recover the image’s 2D structure using its existing 1D position embeddings. Attention is causal throughout — image tokens are not treated specially — and there is no pooling. Because there is no fixed input resolution, images of any aspect ratio and size can be fed in without padding or distortion, and training uses images of varying sizes without staged low-res → high-res phases. This is the load-bearing simplification: architectural components collapse into one trunk, and the “how do we scale the vision side vs. the language side” question disappears because there is no separate vision side to allocate compute to.

The practical cost is Sid’s Slack observation — a full high-resolution UI screenshot at native patch density yields ~30K image tokens, roughly two orders of magnitude more than a ViT-tokenized VLM would emit for the same input. The bet is that (a) the LLM’s attention is expressive enough to absorb this and (b) the simplification pays off in engineering + scaling clarity.

Results

Benchmark table from the post (higher is better; VQAv2 / OKVQA are accuracy%, COCO is CIDEr, AI2D is accuracy%):

Eval	Fuyu-8B	Fuyu-Medium	LLaVA 1.5 (13.5B)	QWEN-VL (10B)	PaLI-X (55B)	PaLM-E-12B	PaLM-E-562B
VQAv2	74.2	77.4	80.0	79.5	86.1	76.2	80.0
OKVQA	60.6	63.1	n/a	58.6	66.1	55.5	66.1
COCO Captions	141	138	n/a	n/a	149	135	138
AI2D	64.5	73.7	n/a	62.3	81.2	n/a	n/a

Fuyu-8B beats PaLM-E-12B on all three shared metrics and beats QWEN-VL on VQAv2 + OKVQA despite being ~2B smaller [Table 1]. PaLI-X (55B, per-task finetuned) is uniformly stronger, framing Fuyu’s numbers as “encoder-free is competitive at parity, not dominant.” Qualitative capabilities showcased in the post: chart / diagram reasoning (multi-hop counting on cast-network diagrams), document + infographic understanding, and — in the internal, non-released Fuyu variants — screen-native bbox_to_text and text_to_bbox for UI agents.

Why it’s interesting

Fuyu is the historical anchor for the encoder-free VLM lineage the wiki has been tracking through 2026. It’s the direct ancestor of Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generation (Tuna-2’s pixel-space UMM with the JiT flow-matching head) and From Pixels to Words: Towards Native Vision-Language Primitives at Scale (NEO), and the scaling-law paper filed in the parent thread (How Far Are We from Removing the Visual Encoder? Scaling Laws for Encoder-Free Multimodal Pretraining) formalizes what Fuyu bet on architecturally in 2023: remove the encoder and the language model will grow its own visual pathways given enough compute. The “how far are we from removing the visual encoder” title reads as a direct callback to Fuyu’s original architectural gamble.

Sid’s ~30K tokens/image note is the exact tradeoff that’s kept the encoder-free direction unpopular for two years: Fuyu compensated for the missing vision prior with sequence length rather than architectural complexity, and only very recently (Tuna-2 masking scheme, NEO-unify MoT, encoder-free scaling laws) has the field started to close that gap with better inductive biases and characterize when it pays. Also worth flagging: Fuyu’s text_to_bbox / bbox_to_text UI-agent capabilities predate the current computer-use agent wave by ~2 years — Adept was building agentic screen-native VLMs before that label existed, and Fuyu is the open artifact of that lineage.

See also

• Unified Multimodal Models — Fuyu is the historical origin of the encoder-free branch of the UMM taxonomy that NEO-unify, Tuna-2, and Inkling later extended.
• How Far Are We from Removing the Visual Encoder? Scaling Laws for Encoder-Free Multimodal Pretraining — the parent-thread scaling-law paper directly answers “does Fuyu’s bet pay off?” — projects the crossover at ~$10^{22}$ FLOPs.
• Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generation — Tuna-2’s pixel-space UMM is Fuyu’s spiritual successor with a masking-based feature-learning scheme and a flow-matching generation head added on top.
• From Pixels to Words: Towards Native Vision-Language Primitives at Scale — NEO, the 2025 encoder-free VLM architecture directly downstream of Fuyu’s decoder-only-with-patch-projection recipe.
• Scaling Native Multimodal Pre-Training From Scratch — MoE encoder-free UMM with decoupled isoFLOP scaling — the “how many parameters does an encoder-free model want” question Fuyu punted on in 2023.
• NEO-unify: Building Native Multimodal Unified Models End to End — NEO-unify at 2B extends the encoder-free recipe to unified generation (no VE and no VAE).
• VLM Perception Failures — Fuyu documented VQA scoring pathologies (correct answers scoring 0) three years before the current wave of VLM-perception-failure benchmarks; the concept page’s evaluation-hygiene thread has a Fuyu-era precedent worth crediting.