Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generation

TL;DR

Tuna-2 is a 7B native unified multimodal model (UMM) from Meta AI / HKU / Waterloo that performs both understanding and generation directly on raw pixels — no VAE, no pretrained vision encoder, just patch-embedding layers feeding a Qwen2.5-7B decoder, with a flow-matching head doing pixel-space x-prediction for generation. The paper runs a controlled comparison between encoder-free Tuna-2 and an intermediate Tuna-R variant that keeps a SigLIP-2 representation encoder, and finds Tuna-2 is competitive on generation (GenEval 0.87) and better on understanding (GQA 65.0, MMVP 77.3, CountBench 81.7), especially on fine-grained / pixel-centric tasks. A masking-based feature-learning scheme — randomly mask patches at the input, predict clean pixels for masked + unmasked regions — is required to stabilize pixel-space training. Headline caveat: at 7B Tuna-2 underperforms 14B BAGEL on most understanding benchmarks; the authors frame this as a parameter-parity issue, which is the framing Kyle’s Slack note picks up on.

Key claims

Tuna-2 removes both the VAE and the pretrained representation encoder, using only patch-embedding layers in front of the LLM decoder; image generation is pixel-space flow matching with x-prediction and velocity regression à la JiT [§2.1, Fig. 1].
A masking-based feature-learning scheme (replace random input patches with a learnable mask token; predict clean pixels at masked + unmasked positions for generation; predict text under partial visual observation for understanding) is the load-bearing trick that makes pixel-space unified training stable [§2.2, Fig. 3].
After sufficient pretraining, the encoder-free Tuna-2 beats the encoder-based Tuna-R on understanding while staying competitive on generation; Tuna-R converges faster early in pretraining but Tuna-2 catches up and overtakes at scale [§1, §3.5].
Among 7B-scale native UMMs, Tuna-2 sets SOTA on GQA (65.0), MMVet (51.7), MMVP (77.3), AI2D (79.6), OCRBench (79.7), CountBench (81.7), and VisuLogic (28.8); pixel-centric benchmarks (V*, CountBench, VisuLogic) are where pixel-space representations help most [Table 1].
Tuna-2 reaches GenEval 0.87 and DPG-Bench 86.54, comparable to Tuna (0.90 / 86.76) and BAGEL-14B (0.88 / 85.07) despite running fully in pixel space [Table 3.1].
LLM-judge (GPT-5.4, Claude Opus 4.7) head-to-head on 1.5K prompts × 4 images: Tuna-2 wins on diversity by a wide margin (48.4% / 41.9% vs Tuna-R’s 30.9% / 29.9%) while matching Tuna-R on quality [Table 3.1].
Image reconstruction probe on ImageNet val: Tuna-R reaches rFID 0.12 / PSNR 32.22 and Tuna-2 reaches rFID 0.15 / PSNR 32.80 at 512², beating all unified tokenizers (UniTok, TokenFlow, X-Omni, MingTok, RAE, PS-VAE) and approaching FLUX.1-dev’s specialized VAE (rFID 0.06 / PSNR 33.65) — direct evidence that the pixel-space unified representation can carry enough information for high-fidelity reconstruction without a dedicated autoencoder [Table 5].
Image editing on ImgEdit: Tuna-2 totals 4.09 — behind FLUX.1 (4.00), Qwen-Image (4.27), GPT-Image (4.20), Tuna (4.31), and Tuna-R (4.18), with the encoder-based Tuna-R slightly ahead, suggesting pretrained visual priors still help fine-grained editing fidelity [Table, ImgEdit].
Underperforms 14B BAGEL on most understanding benchmarks (e.g. MMMU 50.7 vs 55.3, MMVP 77.3 vs 85.0, AI2D 79.6 vs 89.2) — at 7B vs 14B the parity caveat applies [Table 1].

Method

The architecture is deliberately the removal of components rather than the addition of any. Tuna-R is the intermediate halfway-house: drop the VAE, keep SigLIP-2 So400M as a representation encoder feeding a connector → Qwen2.5-7B decoder → either an LM head (for text) or a flow-matching head (for pixel-space image generation). Tuna-2 then drops SigLIP-2 too, replacing it with a simple patch-embed layer; the LLM decoder is the only vision module. Generation uses rectified flow with linear schedule, the model predicts the clean image $\hat{x}_0$ which is converted to a velocity for the flow loss (the JiT recipe). Inference runs Euler steps on the velocity.

The masking scheme is the only architectural addition: randomly sample input patches at a fixed ratio, replace them with a learned mask token, and require the model to predict clean pixels everywhere (not just at masked positions) for generation and to answer text questions under partial visual observation for understanding. The authors connect this to MAE / SigLIP 2 / MaskGIT / DeTok and show empirically that it lifts pretraining-stage metrics.

Training is two-stage and fully end-to-end (no separate connector-pretrain phase, because there’s no connector): Stage 1 pretrains the whole model for 300k steps on 64 nodes on 550M image-text pairs (70/30 caption/T2I split) plus 20% Nemotron text-only, at LR 2e-4. Stage 2 SFT runs 50k steps at LR 2e-5 on FineVision (13M, instruction-following), OmniEdit (~2M, editing), and high-quality T2I data. Sequence length is padded to 16k tokens/GPU.

Results

Headline numbers (all on Qwen2.5-7B-Instruct backbone):

Understanding, general — GQA 65.0, RealWorldQA 67.7, MMVet 51.7, MMMU 50.7, SEED-Bench2+ 61.1 [Table 1]. Tuna-2 ≥ Tuna-R on most metrics, both ≫ Tuna and most other 7B native UMMs.
Understanding, pixel-centric — MMVP 77.3, AI2D 79.6, ChartQA 85.6, OCRBench 79.7, V* 59.2, CountBench 81.7, VisuLogic 28.8 [Table 1]. This is where the encoder-free design pays — Tuna-2 leads all 7B native UMMs on CountBench / VisuLogic and ties at the top on OCRBench.
Generation — GenEval overall 0.87 (vs Tuna 0.90 / Tuna-R 0.88 / BAGEL 0.88), DPG-Bench 86.54 [Table 3.1].
LLM-judge head-to-head — under GPT-5.4 Tuna-2 wins 32.1% quality / 48.4% diversity vs Tuna-R’s 35.7% / 30.9% and Tuna’s 22.3% / 20.6%; under Claude Opus 4.7, 34.8% / 41.9% vs 37.2% / 29.9% vs 28.1% / 28.2% [Table 3.1]. Tuna-2 trades a small amount of quality for a large diversity gain.
Reconstruction — rFID 0.15, PSNR 32.80, SSIM 0.93 at 512² ImageNet val [Table 5] — first or second among unified tokenizers, closing most of the gap to FLUX.1-dev VAE.
Editing — ImgEdit total 4.09 [Table, ImgEdit]; Tuna-R 4.18 slightly ahead, and Tuna 4.31 still better — the only metric where the encoder-based variant has a clear edge.

Against BAGEL-14B (the reference point in Kyle’s Slack note): BAGEL wins MMMU 55.3 vs 50.7, MMVP 85.0 vs 77.3, AI2D 89.2 vs 79.6, VisuLogic 41.7 vs 28.8, but Tuna-2 wins MMVet 51.7 vs 67.2 (BAGEL), CountBench 81.7 vs 82.5 (tied), OCRBench 79.7 vs 73.3, and matches on GenEval (0.87 vs 0.88). The “half the params” defense applies most cleanly to MMMU / VisuLogic (where 14B clearly helps reasoning) and least cleanly to MMVet / OCRBench (where Tuna-2 already wins).

Why it’s interesting

This is the strongest filed datapoint to date for the “remove the pretrained vision encoder and the VAE” thesis in unified multimodal models — a direct architectural sibling to NEO-unify: Building Native Multimodal Unified Models End to End (NEO-unify) but with a controlled ablation against an intermediate encoder-based variant (Tuna-R) on the same backbone and data, which NEO-unify did not publish. The result — pixel-space unified representations help fine-grained understanding (CountBench, V*, VisuLogic) rather than hurt it — is the empirical claim that the wider Unified Multimodal Models page’s “encoder-free is feasible” thread needed to anchor.

It also folds neatly into Pixel-space diffusion: Tuna-2 uses the JiT x-prediction + flow-matching recipe that PixelGen (PixelGen: Improving Pixel Diffusion with Perceptual Supervision) and CAFM (Continuous Adversarial Flow Models) both build on, applied here in the unified setting rather than the generation-only setting those papers studied. The masking-based feature-learning scheme is a different load-bearing trick than PixelGen’s perceptual losses or CAFM’s adversarial criterion — and it would be interesting to see whether they compose.

Kyle’s framing in the Slack thread — “underperform bagel but claim that since theirs is half the params it’s not a fair comparison” — is the right read. The parity defense is partly valid (MMMU / VisuLogic) and partly weak (MMVet / OCRBench where Tuna-2 already wins outright); the cleaner argument is that the Tuna-2 vs Tuna-R ablation, not the Tuna-2 vs BAGEL comparison, is what carries the paper.