Tuna: Taming Unified Visual Representations for Native Unified Multimodal Models

TL;DR

Tuna is a native unified multimodal model (UMM) from Meta BizAI / HKU / Waterloo / KAUST that builds a single continuous visual representation by cascading a 3D causal VAE (Wan 2.2) with a representation encoder (SigLIP 2 with a re-initialized 1×1 patch-embed to match the VAE’s 16× spatial downsampling). The resulting unified features flow into a Qwen2.5 decoder that handles both autoregressive text and flow-matching image/video generation. At 1.5B and 7B scales it sets SOTA across image/video understanding (MMStar 61.2, OCRBench 74.3), image generation (GenEval 0.90, DPG-Bench 86.76), image editing (ImgEdit 4.31 — best among all UMMs), and video generation (best among UMMs on VBench), beating decoupled designs like BAGEL-14B and Mogao at half the parameters. The paper’s central thesis: unified visual representations avoid the spatial/temporal/channel format mismatch between separate semantic and VAE encoders, and joint understanding+generation training in this space makes the two tasks help rather than fight.

Key claims

Tuna’s unified visual representation is built by feeding the 3D causal VAE latent of Wan 2.2 (16× spatial / 4× temporal downsampling) into a modified SigLIP 2 encoder whose 16×16 patch-embed has been replaced with a randomly initialized 1×1 patch-embed, so token-sequence lengths match across the two encoders [§2.2].
Decoupled UMMs like BAGEL and Mogao need MoE-style architectures to handle two incompatible visual encoders (SigLIP at 16× / 1152-d / no temporal vs. Wan VAE at 8× / 16-d / 4× temporal); a unified representation removes this cost and the format-mismatch problem [§1].
For video, the SigLIP-2 features run with a window-based attention by reshaping the 4-latent-frame dimension into the batch dimension, so the representation encoder never sees an excessively long flat sequence [§2.2, Eq. 1–3].
The model uses an autoregressive next-token LM head for text and a flow-matching head (DiT-style with AdaLN-Zero timestep conditioning, multimodal 3D-RoPE over the concatenated text+visual sequence) for visual generation; causal attention over text, bidirectional over visual [§2.2, Fig. 3].
Three-stage training: (1) pretrain the representation encoder + flow head with the LLM frozen on captioning + T2I; (2) unfreeze the LLM and continue pretraining, then introduce instruction-following, editing, and video-captioning data; (3) SFT on a curated mix at LR 2e-5 [§2.3].
At 1.5B, Tuna beats Show-o2-1.5B across all nine reported understanding benchmarks (e.g. MMStar 54.6 vs 43.4, OCRBench 71.9 vs 24.5, ChartQA 82.1 vs 40.0) and outperforms many composite UMMs and understanding-only models [Table 2.3].
At 7B, Tuna sets the highest scores among 7B-scale native UMMs on every reported understanding benchmark (MMStar 61.2 vs Show-o2 56.6, ChartQA 85.8 vs 78.6, OCRBench 74.3 vs 32.4) — and beats BAGEL-14B on most despite half the parameters [Table 2.3].
On GenEval, Tuna-1.5B hits 0.88 overall (beating Show-o2-7B at 0.76 and Janus-Pro-7B at 0.80) and Tuna-7B reaches 0.90, on par with Mogao-7B (0.89) and BAGEL-14B (0.88) [Table 3.1].
On DPG-Bench and OneIG-Bench, Tuna leads UMMs and lands close to Qwen-Image-20B and FLUX.1 Dev; the largest delta is on OneIG-Bench text rendering where Tuna-7B hits 0.82 vs Show-o2-7B’s 0.00 [Table 3.2] — direct evidence that the unified representation carries strong semantic information for visual-text generation.
On ImgEdit-Bench, Tuna-7B scores 4.31 overall, the highest among all evaluated UMMs and competitive with generation-only Qwen-Image-20B (4.27) and FLUX.1 Kontext Pro (4.00) [ImgEdit table].
On video understanding (Tuna-1.5B, 49 frames), Tuna beats Show-o2-1.5B on MVBench (54.4 vs 49.8), Video-MME (49.1 vs 48.0), LongVideoBench (49.7 vs 49.2), and matches LLaVA-OV-7B at 26.9 → 27.4 on LVBench despite being ~5× smaller [Table on video understanding].
On VBench text-to-video, Tuna-1.5B achieves SOTA among UMMs capable of video generation while using only a 1.5B LLM decoder [Table on VBench].
The paper’s ablation argues the cascaded VAE→representation-encoder design beats Show-o2-style late-fusion of SigLIP+VAE features, which the authors find empirically biases the learned representation toward semantics and limits generation quality [§3.4].

Method

The architecture is conceptually simple: two encoders chained, not fused. An input image or video $x$ goes through Wan 2.2’s 3D causal VAE encoder to produce a latent $z$ . For training, a noisy latent $z_t = (1-t)z + t\epsilon$ is sampled (timestep $t$ uniform during generation, fixed to $t=0$ for understanding). The VAE latent then enters a modified SigLIP 2 — same backbone, but the original 16×16 patch-embed is swapped for a randomly initialized 1×1 patch-embed so that the post-VAE 16× downsampling is not re-applied. A 2-layer MLP projects the result into the LLM’s hidden dimension.

For video, the representation encoder operates per 4-frame latent window: the temporal dimension is reshaped into the batch dimension before the SigLIP forward pass and restored afterward, keeping the representation encoder’s effective context length bounded regardless of video length.

The resulting visual tokens are prefixed with a learned timestep token, concatenated with the language tokens, and fed into a Qwen2.5-1.5B-Instruct or Qwen2.5-7B-Instruct decoder. The decoder uses causal attention over text and bidirectional over visual, and shares its weights across understanding and generation. Two output heads peel off the decoder hidden states: the standard Qwen LM head for text, and a randomly initialized DiT-style flow-matching head with AdaLN-Zero conditioning for visual generation. The model uses multimodal 3D-RoPE over the joint text-visual sequence to handle interleaved instructions.

Training is three stages. Stage 1 freezes the LLM and trains only the representation encoder and flow head on a mix of image captioning and text-to-image generation at LR 1e-4, base resolution 512×512. Stage 2 unfreezes the LLM after a 2k-step linear warmup and continues at the same LR, adding instruction-following, image editing, and video-captioning data partway through. Stage 3 is SFT at LR 2e-5 on curated instruction-following + editing + high-quality T2I/T2V data. The 7B variant skips video training due to compute.

Results

Image understanding (Table 2.3). Tuna-1.5B vs Show-o2-1.5B: MMStar 54.6 vs 43.4 (+11.2), OCRBench 71.9 vs 24.5 (+47.4), ChartQA 82.1 vs 40.0 (+42.1), AI2D 71.4 vs 69.0, SEED 69.3 vs 65.6. Tuna-7B sets the top score among 7B native UMMs on every metric.
Image generation (Table 3.1). GenEval overall: Tuna-1.5B 0.88, Tuna-7B 0.90 — the latter the highest filed UMM result on the page along with Mogao-7B (0.89). Subscores show Tuna-7B leading on Position (0.88) and Color Attr. (0.83) where Mogao-7B reports 0.84 / 0.80. DPG-Bench: Tuna-7B 86.76 (beating BAGEL-14B 85.07, Show-o2-7B 86.14). OneIG-Bench overall: Tuna-7B 0.50 vs BAGEL-14B 0.36, Show-o2-7B 0.31; Text rendering Tuna-7B 0.82 vs Show-o2-7B 0.00 — the most striking subscore in the paper [Table 3.2].
Image editing (Table 3.2). ImgEdit-Bench overall: Tuna-7B 4.31, top among UMMs and competitive with FLUX.1 Kontext Pro (4.00), Qwen-Image (4.27). GEdit-Bench G-Overall: Tuna-7B 7.29 vs BAGEL-14B 6.52, OmniGen2-4B 6.41.
Video understanding. Tuna-1.5B at 49 frames: MVBench 54.4, Video-MME 49.1, LongVideoBench 49.7, LVBench 27.4 — beats Show-o2-1.5B on three of four and is within 1 point of LLaVA-OV-7B on MVBench and LVBench despite being ~5× smaller.
Video generation. Tuna-1.5B on VBench: highest among UMMs reported in the table, only the 1.5B LLM decoder is used.

Why it’s interesting

This is the predecessor to Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generation, which is already filed: same lab (Zhiheng Liu, Shoufa Chen, Sen He, Jonas Schult, Wenhu Chen, Ping Luo, Tao Xiang, Yuren Cong) and same Tuna codename, with Tuna-2 going one step further and removing both the VAE and the SigLIP-2 representation encoder, replacing them with a bare patch-embed and a masking-based feature-learning trick. Reading the two papers together is the cleanest way to understand the arc: Tuna says “unified visual representations via VAE→SigLIP cascade beat decoupled SigLIP-and-VAE designs”; Tuna-2 then says “actually, with the right pretraining trick, you don’t even need either encoder.” The Tuna-2 page on the wiki already references Tuna-R as the encoder-based intermediate variant — and Tuna-R is essentially this paper’s architecture, now reading as Tuna-2’s ablation baseline.

Within Unified Multimodal Models, Tuna sits in the AR+Diffusion paradigm with a single shared backbone (vs. the MoT split in NEO-unify, the two-stage MLLM+DiT in DuoGen / OpenUni / BLIP-3-o, or the from-scratch tri-modal MDM in Apple’s design-space paper). What’s distinctive is the cascade over fused-features-style late fusion: Ovis-Image Technical Report and Show-o2 fuse SigLIP and VAE features at the LLM input, while Tuna runs VAE first and SigLIP second, claiming the cascade order avoids the semantic-bias problem the paper attributes to late fusion (§3.4). It also connects to Dual-stream diffusion transformer designs via the AdaLN-Zero flow-matching head and 3D-RoPE; the difference is that Tuna’s “two streams” (text and vision) share decoder weights rather than being separated MMDiT-style.

The OneIG-Bench text-rendering result (0.82 vs Show-o2-7B’s 0.00) is the most surprising number in the paper and worth flagging: it suggests the cascaded VAE→SigLIP representation preserves enough character-level fine detail for visual text, which has historically been the failure mode of VAE-latent generation models. That’s a useful counterpoint to the Pixel-space diffusion thread (PixelGen, CAFM, Tuna-2 itself) which argues pixel-space is required for fine-grained content — Tuna’s evidence is that a properly designed latent UMM also gets there.