Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis

TL;DR

An empirical study of the deep fusion recipe for text-to-image synthesis — a frozen decoder-only LLM and a trainable DiT coupled by layer-wise shared self-attention, so the DiT extracts conditioning from the LLM’s internal hidden states at every layer rather than reading a single text-encoder output. The paper provides controlled apples-to-apples comparisons against self-attention-DiT and cross-attention-DiT shallow-fusion baselines under matched parameters/data, ablates timestep conditioning, positional encoding and base-LLM choice, and proposes FuseDiT — a 2B DiT fused with frozen Gemma-2 2B, trained on 26M images, that reaches GenEval 0.60 / DPG 81.6 / FID 7.54. The interesting bits are negative results: full removal of AdaLN-Zero is the best timestep-conditioning option, instruction-tuned LLMs hurt image-text alignment, and DiT quality is strongly bottlenecked by base-LLM capability.

Key claims

Deep fusion (layer-wise shared self-attention between frozen LLM and trainable DiT) beats both shallow-fusion baselines on image-text alignment (GenEval 0.51 vs 0.42 self-attn / 0.49 cross-attn) at matched ~2.5B parameters and 300K steps; shallow fusion wins on FID [Table 1].
Deep fusion is also the fastest of the three at inference (1.66 s vs 1.75 / 1.86 s) despite using a frozen LLM stream [Table 2].
Completely removing timestep conditioning (no AdaLN-Zero, no AdaLN-Single, no additive timestep embedding) improves FID from 27.33 → 21.27 while keeping GenEval (0.51 → 0.49) and DPG (76.6 → 76.7) within noise — a 20% reduction in total parameters [§6.1, Table 3].
Adding pooled CLIP-L/14 text into AdaLN-Zero (the SD3-style augmentation) slightly improves FID but weakens image-text alignment [§6.1, table after Q1.1].
For positional encoding, 1D-RoPE on text + 2D-RoPE on image gives the best overall numbers; pure 1D-RoPE over the joint sequence underperforms, and Qwen2-VL-style M-RoPE underperforms direct 1D+2D combination [§6.2, Tables 4–5].
Instruction-tuned Gemma 2B IT underperforms base Gemma 2B as the deep-fusion LLM (GenEval 0.49 vs 0.51), and adding an “Imagine: ” instruction prompt partially recovers but does not exceed the base [§6.3, Q3.1].
Multi-modal pretraining (PaliGemma 3B PT vs Gemma 2B) gives a small positive lift (GenEval 0.52 vs 0.51) [§6.3, Q3.2].
A 6-point absolute language-benchmark jump from Gemma 2B → Gemma 2 2B translates to a drastic image-generation lift (GenEval 0.51 → 0.54, DPG 76.6 → 79.1, FID 27.33 → 23.94) — the DiT’s image-generation quality is gated by base-LLM capability [§6.3, Q3.3].
FuseDiT — the recipe combining “no timestep conditioning + 1D+2D RoPE + Gemma 2 2B base + 800K steps on 26M CC12M+SA-1B+JourneyDB images” — reaches GenEval 0.60 / DPG 81.6 / FID 7.54 at 2B parameters, on par with SD3-M and ahead of PixArt-Σ and Lumina-Next at comparable budgets [§7.2, Table 6].
KV states of the frozen LLM stream are constant across diffusion timesteps and can be cached once per prompt, eliminating LLM-side compute during sampling [§3.1].
The deep-fusion architecture is reinterpretable as a special case of self-attention DiT: both aggregate text via in-context self-attention, but deep fusion’s per-layer K/V come from the corresponding LLM layer rather than a trainable projection of the last hidden state [§5.1].

Method

The architecture is a two-stream transformer: a frozen decoder-only LLM stream processing the prompt tokens, and a trainable DiT stream processing noised image latents. At every layer, the two streams share the same self-attention operation — text and image hidden states are concatenated for attention, with a causal mask on text-to-text and bidirectional on image-to-image, and the cross-stream block lets image tokens attend to text tokens but not vice versa (so the LLM behaves identically to its pretrained autoregressive role). After the final layer, text tokens are discarded and only image tokens predict the rectified-flow velocity [§3.1, Fig. 2]. The DiT’s transformer config (hidden size, layers, heads, FFN) is matched layer-for-layer to the base LLM so that every DiT layer has a direct counterpart LLM layer; weights are independent. Training is rectified flow with the SD3 logit-normal timestep distribution, BF16 mixed precision, AdamW, constant LR 1e-4, batch size 512, EMA decay 0.99, 10% CFG dropout, and TPU v4-256 + FSDP via PyTorch/XLA SPMD [§4]. The 16-channel SD3 VAE is reused.

The shallow-fusion baselines for comparison are (a) self-attention DiT — last-layer LLM hidden state projected to K/V and concatenated in self-attention, mirroring Lumina-Next / PG-style; (b) cross-attention DiT — last-layer LLM hidden state projected to K/V and consumed by an additional cross-attention layer per DiT block, mirroring PixArt-α/Σ and Sana [§5.1, Fig. 3]. Both baselines include RMS-norm + linear text preprocessing before the additional projections.

The Sec. 8 follow-up explores aggressive variants the main study does not adopt: fusing a smaller DiT into the middle layers of a larger LLM (since LLM middle layers carry richer semantics, per [39]) so DiT can scale independently of LLM, and the architecture-alignment question — what happens when the DiT hidden width differs from the LLM hidden width and only projects to the LLM’s Q/K/V dimensions for the cross-stream attention.

Results

Controlled deep-vs-shallow comparison (300K steps, CC12M, 2.5B-class models): deep fusion 0.51 GenEval, 76.6 DPG, 27.33 FID; cross-attention 0.49 / 76.3 / 24.00; self-attention 0.42 / 73.9 / 26.16 [Table 1].
Timestep-conditioning sweep, parameter cost vs quality: AdaLN-Zero 2.47B → 0.51 / 76.6 / 27.33; AdaLN-Single 2.01B → 0.47 / 75.2 / 27.09; addition 1.99B → 0.47 / 75.6 / 26.40; no timestep 1.98B → 0.49 / 76.7 / 21.27 [Table 3]. The no-timestep option is the recipe’s chosen default.
FuseDiT (2B params, 26M samples, 800K steps, Gemma 2 2B + 1D+2D-RoPE + no timestep): GenEval 0.60, DPG 81.6, FID 7.54 — surpasses many industry baselines at far smaller data scale (SD 3-M is 2B params on 1B data → 0.62/84.1/11.92; SDXL 2.6B/1.6B → 0.55/74.7/6.63; PixArt-Σ 0.6B/46M → 0.54/80.5/6.15) [§7.2, Table 6].

Why it’s interesting

This is a careful, reproducible empirical study of the frozen-LLM-as-text-encoder recipe that several recent unified-multimodal systems (LMFusion, PG-3, grafting variants) hinge on, but which has been undisclosed in detail. It directly informs the Unified Multimodal Models cluster’s “what’s the cleanest way to bolt a generator onto a frozen MLLM” open question — the answer here is layer-wise shared self-attention with all LLM layers exposed as conditioning, no AdaLN-Zero, 1D+2D RoPE, and do not use the instruction-tuned variant. It also complements Dual-stream diffusion transformer: that page’s three filed papers (LTX-2, MOVA, SkyReels-V4) are dual-stream A/V designs that all keep both streams trainable, while FuseDiT is dual-stream text/image with one stream frozen — an interesting capacity-allocation contrast. The negative result on instruction tuning rhymes with When Pretty Isn't Useful: Investigating Why Modern Text-to-Image Models Fail as Reliable Training Data Generators‘s broader theme that “the dataset/model property you want for downstream use isn’t the one you optimized for upstream.” The strong base-LLM dependence (Gemma 2 jump → big lift) is a concrete data point that text-to-image quality is bottlenecked by LM-side reasoning, not just by the visual decoder — an empirical anchor for the open “does language capability transfer to generation?” line.