Looped Diffusion Transformer

TL;DR

Looped-DiT scales text-to-image diffusion by repeatedly running shared Transformer blocks within each denoising step, trading parameter count for computational depth at fixed parameters. Naive looping fails because of weak supervision across intermediate loops and unregulated attention updates that erode local information; Looped-DiT adds deep supervision across intermediate loops plus a self-modulating attention mechanism to stabilize feature updates. Under matched-parameter and matched-compute budgets, a 260M looped model surpasses a 6.5× larger non-looped baseline across multiple T2I benchmarks while using 4.9× lower inference compute. Beyond iso-compute wins, loop depth outperforms adding more denoising steps under a fixed inference budget, and deeper loops progressively correct earlier mistakes — behavior the authors frame as suggestive of latent reasoning.

Key claims

• Looping a shared Transformer block within each denoising step raises effective computational depth without raising parameter count, scaling T2I quality along a dimension distinct from model size or denoising step count [Abstract].
• Naive looping does not consistently improve image quality; two specific pathologies are identified: weak supervision across intermediate loops, and unregulated attention updates that progressively erode local information [Abstract].
• Deep supervision across intermediate loops combined with self-modulating attention stabilizes looped feature updates and recovers the gains that naive looping fails to deliver [Abstract].
• Under matched-parameter and matched-compute, Looped-DiT consistently beats non-looped baselines, with a 260M-parameter looped model exceeding a 6.5× larger non-looped model on multiple T2I benchmarks at 4.9× lower inference compute [Abstract].
• Under a fixed inference compute budget, increasing loop depth yields larger quality gains than adding more denoising steps [Abstract].
• Deeper loops progressively correct mistakes made in earlier loops, a behavior the authors describe as suggestive of latent reasoning in diffusion [Abstract].

Method

Looped-DiT follows the standard “middle cycle” layout of the looped-transformer family: a shared Transformer block is applied L times per denoising step inside a text-to-image DiT backbone, so effective depth is block-depth × L while unique parameters stay at block-depth. The paper identifies two failure modes of naive looping and addresses each. First, intermediate loops receive no explicit learning signal, so the authors introduce deep supervision applied across intermediate loop outputs rather than only at the final loop. Second, repeated self-attention over the same parameters progressively washes out local information; the authors add self-modulating attention that regularizes how attention updates evolve across loop iterations. The combined recipe trains stably and lets loop count serve as both a train-time and inference-time compute lever.

Results

• A 260M-parameter Looped-DiT surpasses a 6.5× larger non-looped baseline across multiple text-to-image benchmarks while requiring 4.9× lower inference compute [Abstract].
• At fixed inference compute, adding loop depth outperforms adding denoising steps [Abstract].
• Qualitative probes show that deeper loops progressively correct errors visible at earlier loops within the same denoising step [Abstract].

Why it’s interesting

Looped-DiT is the second visual-generation entry in the Looped Transformers cluster after ELT: Elastic Looped Transformers for Visual Generation (ELT), and the first to pin down a diffusion-specific failure mode — unregulated attention washing out local information — that ELT’s ILSD recipe does not explicitly address. Where ELT frames its training trick as “prefix-as-student self-distillation” to make loop count a continuous inference-time knob, Looped-DiT frames it as “deep supervision + self-modulating attention,” and the two prescriptions should be read against each other. The claim that loop depth beats denoising steps under a fixed inference budget is a direct datapoint for the Looped Transformers open question on train-time vs inference-time loop count, and sits on the opposite side of the The Serial Scaling Hypothesis “fixed-depth diffusion is in TC⁰, no amount of steps escapes” argument from the diffusion-step-scaling literature. The “latent reasoning via progressive loop correction” framing echoes Fractal basins trap latent reasoning‘s dynamical-systems account of loop iterations as saddle-point search.

See also

• Looped Transformers — concept page; Looped-DiT is the second visual-generation entry alongside ELT: Elastic Looped Transformers for Visual Generation.
• ELT: Elastic Looped Transformers for Visual Generation — ELT: prior looped T2I/video model; ILSD solves the “weak supervision at intermediate loops” pathology via prefix-as-student rather than deep supervision.
• Hyperloop Transformers — Hyperloop: the LLM-side companion that uses loop-boundary hyper-connections to stabilize weight-shared updates — different stabilization mechanism than Looped-DiT’s self-modulating attention.
• Diffusion training efficiency — looping is a parameter-efficient path to deeper DiTs without more unique weights.
• Inference-Time Scaling — loop depth as an intra-denoising-step compute axis, orthogonal to step count.
• The Serial Scaling Hypothesis — complexity-theoretic argument that fixed-depth diffusion backbones sit in TC⁰; Looped-DiT’s “loop depth beats more steps” result is empirical support for the “real depth, not more steps” side.
• Fractal basins trap latent reasoning — dynamical-systems account of latent reasoning in looped transformers; Looped-DiT’s “progressive error correction across loops” claim is a diffusion-side analog.