Encoder-Decoder Gemma: Improving the Quality-Efficiency Trade-Off via Adaptation

TL;DR

T5Gemma adapts pretrained decoder-only Gemma 2 checkpoints (2B and 9B) plus a suite of newly trained mT5-sized models (Small/Base/Large/XL, up to 1.6B) into encoder-decoder LLMs by initializing both stacks from the decoder-only weights and continuing pretraining under a UL2 or PrefixLM objective. The adaptation recipe lets the encoder and decoder be different sizes (e.g. 9B-2B), giving a new quality-vs-latency knob: a 9B encoder paired with a 2B decoder gets accuracy close to 9B-9B at the latency of 2B-2B. Headline result is that on GSM8K, T5Gemma 9B-9B scores ~9 points higher than the original Gemma 2 9B at similar latency, and T5Gemma 9B-2B beats Gemma 2 2B by a large margin at nearly identical latency. Released as an open checkpoint family on Hugging Face, this is the paper behind Google’s July 2025 T5Gemma blog post.

Key claims

Decoder-only LLMs can be adapted into encoder-decoder LLMs by initializing encoder and decoder parameters from the same pretrained decoder-only checkpoint and then continuing pretraining under UL2 or PrefixLM, rather than training the encoder-decoder from scratch [§Abstract].
The adaptation framework supports unbalanced encoder-decoder pairs (e.g. 9B encoder + 2B decoder), which is a new lever for trading deep input understanding against generation cost on tasks like summarization [§Method, blog summary].
T5Gemma sits on or above the quality–inference-efficiency Pareto frontier of the decoder-only Gemma 2 baselines on SuperGLUE and other benchmarks [§Results].
On GSM8K (math) and DROP (reading comprehension) after pretraining alone, T5Gemma 9B-9B beats Gemma 2 9B by >9 points on GSM8K and ~4 points on DROP [§Results].
After instruction tuning, T5Gemma 2B-2B IT exceeds Gemma 2 2B IT by nearly 12 MMLU points and lifts GSM8K from 58.0% to 70.7% — the adapted architecture responds more strongly to instruction tuning than the decoder-only baseline [§Results].
T5Gemma 9B-2B reaches accuracy close to T5Gemma 9B-9B at latency nearly identical to Gemma 2 2B, demonstrating that the unbalanced configuration delivers a real quality-latency Pareto improvement on GSM8K [§Results].
Adaptation is presented as cheaper than pretraining an encoder-decoder of the same size from scratch — the decoder-only checkpoint’s capability is inherited rather than re-acquired [§Abstract].

Method

Each encoder-decoder T5Gemma variant is built in two stages. First, the encoder and decoder stacks are both initialized from the weights of a pretrained decoder-only Gemma 2 model (or, for the smaller mT5-sized variants, from newly pretrained decoder-only models at Small/Base/Large/XL). Second, the resulting encoder-decoder is continued-pretrained under one of two text-to-text objectives: UL2 (the T5/UL2 span-corruption mixture-of-denoisers) or PrefixLM (causal LM with bidirectional attention over a prefix). The paper systematically explores both objectives and various parameter initialization / optimization choices.

A key architectural degree of freedom is that the encoder and decoder are not required to have the same depth/width — a 9B encoder can be paired with a 2B decoder, giving the “9B-2B” configuration. Evaluation covers SuperGLUE (representation quality), GSM8K (math reasoning), DROP (reading comprehension), MMLU, and a quality-vs-latency frontier on GSM8K specifically. The blog release adds the matrix of public checkpoints: pretrained and instruction-tuned variants for 2B-2B, 9B-9B, 9B-2B, plus T5-sized Small/Base/Large/XL, in both UL2 and PrefixLM flavors.

Results

Headline quantitative claims (taken from the Google Developers blog post summarizing the paper):

Pretraining gains over Gemma 2 9B: T5Gemma 9B-9B is >9 points higher on GSM8K and ~4 points higher on DROP.
Instruction-tuning amplification: T5Gemma 2B-2B IT scores ~12 points higher than Gemma 2 2B IT on MMLU; GSM8K rises from 58.0% to 70.7%.
Unbalanced configurations: T5Gemma 9B-2B reaches accuracy close to 9B-9B on GSM8K at near-identical latency to Gemma 2 2B.
Pareto frontier: T5Gemma “nearly dominates” the quality-inference-efficiency Pareto frontier vs decoder-only Gemma counterparts on the reported benchmarks, with SuperGLUE highlighted as the representation-quality witness.

The release ships pretrained and instruction-tuned checkpoints under the Gemma license, hosted on Hugging Face (e.g. google/t5gemma-2b-2b-ul2, google/t5gemma-9b-9b-prefixlm).

Why it’s interesting

The result is unusual: most of the open-frontier LLM landscape filed in the wiki is decoder-only, and the encoder-decoder family had largely been written off as a 2022-era design. T5Gemma resurrects it as a deployment-time lever — the 9B-2B configuration gets close to 9B-9B accuracy at 2B-2B latency, which is a different shape of speed-up from the inference-efficiency methods catalogued under LLM Inference Efficiency (speculative decoding, KV-cache compression, weight quantization). Those all preserve the decoder-only stack and try to make it faster to invoke; T5Gemma changes the stack itself. Complements Mercury: Ultra-Fast Language Models Based on Diffusion (diffusion LLMs as another non-AR architecture for inference speed) and the model-conversion thread in ZAYA1-8B-Diffusion-Preview: Converting an AR MoE to Diffusion on AMD (converting a pretrained AR MoE into a diffusion model) — both are precedents for “adapt a pretrained AR model into a different architecture rather than retraining from scratch.” The follow-up T5Gemma 2 extends the same recipe to multimodal Gemma 3 with tied embeddings and merged self/cross-attention; worth filing separately if it shows up.