T5Gemma 2: Seeing, Reading, and Understanding Longer

TL;DR

T5Gemma 2 extends the T5Gemma decoder-only → encoder-decoder adaptation recipe from text to multimodal (text + image) and long context (up to 128K), starting from the Gemma 3 checkpoints at 270M, 1B, and 4B per stack. Two efficiency tweaks: (1) tie all word embeddings across encoder input, decoder input, and decoder output, saving ~10.5% of params with no quality loss; (2) merge decoder self- and cross-attention into a single joint module with shared weights, saving another ~6.5% at a small (~0.3 pt) quality cost. Pretrained on 2T UL2 tokens with input/output sequences up to 16K each, the resulting models match or beat their Gemma 3 counterparts in pretraining and clearly outperform them after light post-training — and crucially, they generalize from 16K training context to 128K evaluation context, where decoder-only Gemma 3 collapses on RULER and the original (text-only) T5Gemma scores near zero. Released as open pretrained checkpoints at 270M-270M, 1B-1B, and 4B-4B.

Key claims

Adapting a pretrained decoder-only LLM into an encoder-decoder LLM via UL2 continued pretraining (the T5Gemma recipe) generalizes from text-only Gemma 2 to multimodal Gemma 3 — even when the source decoder-only model (Gemma 3 270M, Gemma 3 1B) is itself text-only, the adapted encoder-decoder picks up non-trivial multimodal capability [§1, §4 Text-only LLMs can be adapted into strong multimodal and long-context encoder-decoder models].
Tying all word embeddings (encoder input + decoder input + decoder output/softmax) reduces total parameters by 10.5% with essentially no quality change (47.8 → 47.7 average) [Table 1].
Merging decoder self- and cross-attention into a single attention sub-layer with shared weights — concatenating encoder output and decoder self-attn input before a joint softmax — saves 6.5% of params at a 0.3-point average quality cost [Table 1, Eqs. 1–5].
A rejected ablation: applying cross-attention only on global-attention decoder layers (1-in-6 in Gemma 3’s 5:1 local:global pattern) drops quality by 1.3 points [Table 1].
The encoder and vision encoder are frozen-or-fed-once: a 400M SigLIP encoder turns an image into 256 tokens, which are routed to the encoder (not the decoder) and all encoder tokens see each other bidirectionally [§2].
Long-context training uses positional interpolation on RoPE with separate base frequencies for local (10K) and global (1M) attention layers, following Gemma 3 [§2].
T5Gemma 2 was pretrained on 2T UL2 tokens at 16K input × 16K target sequence lengths, with five span-corruption denoisers (mean span × corruption rate combinations) mixed 1:1:1:1:4 for text and prefix-LM for vision data; no distillation [§3.1].
Distillation is no longer worth the data-loading overhead at this scale — UL2-only (39.8 avg) is within 0.4 points of UL2+KD (40.2) at 1B-1B and beats PrefixLM+KD (38.8) [Table 3].
T5Gemma 2 1B-1B reaches multimodal average 49.8 and long-context average 43.8 — within 8.7 / 6.9 points of the much larger Gemma 3 4B (58.5 / 50.7) [Table 4].
After post-training, T5Gemma 2 4B-4B beats Gemma 3 4B on every benchmark category measured: reasoning (37.7 vs 35.1), STEM+code (63.1 vs 60.9), multilingual (53.0 vs 50.2), multimodal (63.7 vs 60.3), and long-context (62.5 vs 50.1) [Table 5].
On RULER-128K, T5Gemma 2 4B-4B reaches 57.6 (pretraining) and 39.5 (post-training) versus the original T5Gemma 9B-9B at 0.5 — a >100× improvement that the authors attribute to the encoder-decoder architecture extrapolating beyond its 16K training context [Table 4, Table 5].
The downstream embedding model EmbeddingGemma is built on T5Gemma 2 checkpoints to achieve SOTA on text retrieval, identified as a use case where the encoder-decoder design pays off for representation extraction [§5].

Method

T5Gemma 2 is an encoder-decoder Transformer whose building blocks mirror Gemma 3: grouped-query attention with QK-norm, pre- and post-RMSNorm, RoPE, and interleaved local:global attention at a 5:1 ratio. Both encoder and decoder are initialized from the same Gemma 3 pretrained checkpoint (text-only at 270M/1B, multimodal at 4B), then continued-pretrained with UL2 for text and prefix-LM for vision data. A frozen 400M SigLIP encoder converts images to 256 tokens that are fed to the encoder side, where all tokens have bidirectional visibility. Two architectural changes from T5Gemma: (a) all word embeddings — encoder input, decoder input, and decoder softmax — are tied, cutting embedding parameters roughly in half; (b) the decoder’s self-attention and cross-attention sub-layers are merged into a single joint attention by concatenating the encoder output with the decoder self-attention input before a unified softmax with shared K/V projections. Long context uses RoPE positional interpolation with separate base frequencies (10K local / 1M global). Three sizes ship: 270M-270M, 1B-1B, 4B-4B, all pretrained on 2T tokens at 16K × 16K input/output lengths.

Results

Architecture ablations [Table 1]: baseline 47.8 avg @ 4417M (1180M non-embed) → tied embedding 47.7 @ 4417M (590M) → merged attention 47.5 @ 4049M (1180M). Cross-attention only on global layers drops 1.3 points and is rejected.
Pretraining vs Gemma 3 [Table 4]: T5Gemma 2 outperforms Gemma 3 at 270M and 1B scales across reasoning, STEM+code, and multilingual averages. At 4B-4B vs Gemma 3 4B: reasoning 60.8 vs 62.4, STEM+code 33.2 vs 32.8, multilingual 51.4 vs 49.6, multimodal 59.2 vs 58.5, long-context 57.1 vs 50.7.
Long-context extrapolation [Tables 4, 5]: trained on 16K, T5Gemma 2 4B-4B scores RULER-128K 57.6 (pretraining) and 39.5 (post-training); the original text-only T5Gemma 9B-9B scores 0.5 — i.e., it collapses on long-context despite being more than twice as large.
Post-training [Table 5]: T5Gemma 2 4B-4B beats Gemma 3 4B on every category — including MMMU 47.8 vs 47.3, AI2D 78.3 vs 75.0, ChartQA 75.0 vs 69.0, and MRCR 128K 57.7 vs 41.9 — despite using only distillation post-training (no RL), positioned as a lower bound.
Distillation no longer matters at this scale [Table 3]: at 1B-1B, UL2 (39.8) ≈ UL2+KD (40.2) ≫ PrefixLM+KD (38.8); the authors drop distillation for the released checkpoints.
Text-only → multimodal lift: T5Gemma 2 270M-270M and 1B-1B inherit non-trivial multimodal capability despite their Gemma 3 base models being text-only (e.g., 1B-1B reaches DocVQA 66.6, ChartQA 50.2, MMMU 28.4 after pretraining) — the SigLIP encoder + UL2 adaptation supplies the vision skill that the base lacks.

Why it’s interesting

This is the direct multimodal+long-context follow-up to Encoder-Decoder Gemma: Improving the Quality-Efficiency Trade-Off via Adaptation (T5Gemma 1), filling the “blind, short-context” gap the first paper had against decoder-only frontier LLMs. The most striking number is RULER-128K: trained on 16K, T5Gemma 2 4B-4B holds at 57.6 while the original (text-only, larger) T5Gemma 9B-9B drops to 0.5 — long-context extrapolation appears to be a real property of the encoder-decoder architecture when paired with positional interpolation, not just a side-effect of scale. This is a different angle on long-context than the wiki has filed before: VideoNSA: Native Sparse Attention Scales Video Understanding and Captain Safari: A World Engine with Pose-Aligned 3D Memory make the long-context attention pattern sparser; Memory in the Age of AI Agents and the Parametric memory cluster offload context into weights or external stores; T5Gemma 2 keeps standard dense attention and just changes who reads what. Also adds another datapoint to the “adapt rather than retrain” pattern alongside ZAYA1-8B-Diffusion-Preview: Converting an AR MoE to Diffusion on AMD (AR MoE → diffusion). The merged-attention trick is worth a closer look — collapsing self+cross into one softmax with shared K/V mirrors how decoder-only LLMs already work, and makes the encoder-decoder decoder essentially a Gemma 3 decoder with extra input tokens, which is why initialization from a decoder-only checkpoint composes cleanly.