Skip to content
Bud's Wiki

Looped Transformers

TL;DR

Section titled “TL;DR”

Looped Transformers share parameters across depth by applying the same block of layers iteratively rather than stacking unique layers, decoupling effective depth from unique-parameter count. The 2026 wave of work argues that with the right training trick (intra-loop self-distillation, hyper-connections at loop boundaries) and the right block layout (“middle cycle”: loop only an inner block, keep dedicated begin/end blocks), looped models can match a 2×-parameter non-looped Transformer at fixed compute — across visual generation, LLM pretraining, and quantization-aware deployment. Loop count is emerging as a distinct scaling axis with its own predictable scaling-law structure and as a deployment-time compute lever (any-time inference).

Key claims

Section titled “Key claims”
  • A weight-shared block of L_b layers applied L times has the same effective depth L_b·L as a non-looped Transformer with L_b·L unique layers, but only L_b unique parameters — separating representational capacity from parameter count (ELT: Elastic Looped Transformers for Visual Generation §4).
  • Vanilla looped Transformers trained at a fixed loop count L underperform depth-matched non-looped baselines on perplexity at the LLM scales tested (14.85 vs 14.65 at 240M-class) and produce uninterpretable intermediate-loop representations unless explicitly trained to be exit-friendly (Hyperloop Transformers §4.2 Table 1; ELT: Elastic Looped Transformers for Visual Generation §4 / Fig. 8(b)).
  • The “middle cycle” recipe — dedicated begin / middle / end blocks where only the middle block is looped — is the consensus layout across both visual-generation and LLM looped models, and is the layout adopted by both ELT and Hyperloop (Hyperloop Transformers §3; ELT: Elastic Looped Transformers for Visual Generation §4).
  • Two complementary recipes recover or exceed non-looped quality at the same loop count. (a) Intra-Loop Self Distillation (ILSD): stochastically sample an early-exit student trajectory, supervise it with both ground truth and the (stop-gradded) full-L teacher; because the student is a literal prefix of the teacher, marginal cost is just the student’s loss (ELT: Elastic Looped Transformers for Visual Generation §4 / Algorithm 1). (b) Loop-boundary hyper-connections: expand the residual stream into n parallel streams between loop iterations only, using a data-dependent diagonal mixing matrix — outperforms both vanilla Looped and full mHC at fewer params (Hyperloop Transformers §4.3 Tables 5–6).
  • A two-stack hierarchical-recurrence variant is a parallel branch within the looped-transformer family: rather than looping a single block L times, run two distinct Transformer stacks (slow H, fast L) and iterate the L-stack faster than the H-stack with additive cross-level state injection (HRM-Text: Efficient Pretraining Beyond Scaling §Method). Open at LLM scale whether the hierarchical-recurrence variant is strictly dominated by single-stack looped + hyper-connections or whether the multi-timescale split has independent value.
  • The Recursive Reasoning Model (RRM) framing covered by the deterministic papers above (ELT: Elastic Looped Transformers for Visual Generation, Hyperloop Transformers, HRM-Text: Efficient Pretraining Beyond Scaling) commits to a single latent trajectory and a single prediction. A generative / stochastic branch lifts this restriction: GRAM models the recursive latent as a stochastic trajectory trained by amortized variational inference, supporting both conditional reasoning pθ(y∣x)p_\theta(y\mid x) and unconditional generation pθ(x)p_\theta(x), and reports improvements over deterministic recurrent and recursive baselines on structured reasoning and multi-solution constraint satisfaction (Generative Recursive Reasoning §Abstract).
  • Complexity-theoretic motivation for the cluster. Fixed-depth Transformers, SSMs/Mamba, and even fixed-backbone diffusion models with infinite denoising steps all sit inside TC⁰ — they cannot solve inherently serial problems regardless of width or step count. Layer-repeating / looped architectures, RNNs, and autoregressive Chain-of-Thought are the only listed model classes that escape TC⁰ (The Serial Scaling Hypothesis §2.3, Theorem 2.4, Table 2). This is the formal reason looping (or CoT, or recurrence) recovers capability that no amount of width can buy.
  • Loop count has predictable scaling-law structure as a training-time axis. For a fixed parameter budget, data and loop count should be scaled in tandem (Layer-looping scaling laws — Hayden Prairie tweet). Parameter scaling is power-law in unique-layer count L_b across model widths (ELT: Elastic Looped Transformers for Visual Generation §5.2 / Fig. 5).
  • Loop count also acts as an inference-time compute lever (“any-time inference”). ILSD-trained models traverse a near-flat quality curve from l=1 to l=L; UCF-101 models trained with L=12 reach their peak FVD (69.20) at inference L=24, exceeding their training depth (ELT: Elastic Looped Transformers for Visual Generation §5.4 / Fig. 7). In the stochastic regime, inference-time scaling decomposes into two axes: recursive depth × number of parallel trajectory samples (Generative Recursive Reasoning §Abstract).
  • Iso-parameter / iso-compute, looped models can match a 2× larger non-looped Transformer. ELT-XL (111M) matches MaskGIT-XL (446M) at FID 2.0 on ImageNet-256 (ELT: Elastic Looped Transformers for Visual Generation Table 1); 1.1B ELT(Lb=16,L=2L_b{=}16, L{=}2) reaches FID 2.83 vs 3.43 for a 2.1B 32-layer DiT (ELT: Elastic Looped Transformers for Visual Generation §5.2); Hyperloop matches the 240M / 1B / 2B Transformer baselines on PPL with roughly half the parameters (Hyperloop Transformers Table 1). HRM-Text pushes the matching-via-recurrence claim into the pretraining-token-efficiency regime: 1B HRM-Text reaches 60.7% MMLU / 81.9% ARC-C / 84.5% GSM8K with 100–900× fewer tokens than 2–7B open baselines (HRM-Text: Efficient Pretraining Beyond Scaling §Abstract).
  • A minimum unique-block size matters: L_b=1 looped a lot (e.g. L_b=1, L=32) collapses regardless of effective depth, so the looped block must be deep enough to be individually expressive (ELT: Elastic Looped Transformers for Visual Generation §5.2).
  • Looping gains survive INT4 weight-only post-training quantization. Despite the concern that the shared block sees “more” effective inputs and would be harder to quantize, Hyperloop’s INT4 perplexity remains below the dense Transformer’s INT4 baseline at all three scales (Hyperloop Transformers Table 1).
  • Letting loop weights deviate slightly across iterations helps. ELT achieves this implicitly via ILSD’s per-loop supervision; Hyperloop achieves it explicitly via per-loop hyper-connection parameters and a learned loop-position embedding. Direct comparison with LoRA-on-loops (relaxed Recursive Transformers): a rank-32 LoRA per loop (143M params) underperforms Hyperloop at 136M params (14.77 vs 14.40 PPL), so the HC route is the more parameter-efficient way to break weight-tying (Hyperloop Transformers Table 7).
  • Looped models support early-exit / dynamic-depth inference. Logit-lens analysis shows residual streams are more aligned with the vocabulary distribution at each loop boundary than in a non-looped Transformer of matched depth (Hyperloop Transformers §4.4 / Fig. 4).
  • Stabilizing deep recurrence at 1B+ language-model scale required new normalization and gradient-warmup tricks that single-stack looped LLM work has not needed at the same scale: MagicNorm and warmup deep credit assignment (HRM-Text: Efficient Pretraining Beyond Scaling §Abstract). Open whether the hierarchical (two-stack) variant is the reason these are necessary, or whether single-stack looped LLMs would benefit if pushed to the same depth × token budget.
  • Iterative refinement as production design pattern. OnePiece carries the “iteratively refine a representation with cached context” idea into industrial cascade ranking: K reasoning steps each emit a block of M tokens that attend to the cached backbone context via KV-caching, with reasoning bandwidth controlled by block size M (analogous to loop count L) and progressive per-block supervision aligned to the user-feedback funnel (exposure → click → order). Ranking-side block-size scaling is monotonic with diminishing returns past M≈12, and online A/B reports +1.12% GMV/UU + 2.90% advertising revenue at Shopee scale (OnePiece: Bringing Context Engineering and Reasoning to Industrial Cascade Ranking System §4–§7).

Recent contributions

Section titled “Recent contributions”
  • [2026-07-15] Three things everyone should know about Vision Transformers: Shallow-wide dual to the deep-narrow loop-transformer direction: parallel ViT reorganizes 2N sequential MHSA+FFN blocks into N layers of two parallel MHSA+FFN pairs at matched parameters and FLOPs, halving effective depth. ViT-B18×2 matches ViT-B36×1 on ImageNet (83.9 vs 83.9) with 45% faster batch-1 latency on V100 — an upper bound on how much accuracy is on the table from widening without adding parameters, complementing ELT: Elastic Looped Transformers for Visual Generation and Hyperloop Transformers which trade the opposite direction.
  • [2026-06-27] Winfree Oscillatory Neural Network: WONN — first non-transformer entry on this page. Replaces the inner attention block with separable Winfree phase-coupling dynamics but keeps the looped-transformers scaffold (T parameter-shared recurrent updates per layer, dual phase/frequency state with fast-inner/slow-outer timescales). Matches HRM on Maze-hard at 1.47% the params (76.2% vs 74.5% at 0.396M vs 27M), and energy voting (sample 32 trajectories, select by interaction-energy Lyapunov function) lifts to 80.1% — a deterministic-dynamics analog of GRAM’s stochastic parallel-trajectory inference scaling. Also the first synchrony-based architecture reported competitive on ImageNet-1K (76.78% at 12.28M params, matching ResNet-50 at half the params).
  • [2026-05-28] DiffusionBlocks: Block-wise Neural Network Training via Diffusion Interpretation: DiffusionBlocks (Sakana AI, ICLR 2026) reinterprets each iteration of a shared looped block as a denoiser for a specific noise-level range under the EDM noise prior; training becomes K independent denoising problems with no inter-block gradients, replacing BPTT-through-L-iterations with a single forward pass. Directly addresses the cluster’s open question about training cost at depth — the announce thread lists recurrent-depth (looped) transformers as one of five validated architecture families (alongside ViT, DiT, masked-diffusion, autoregressive). Open: no LLM-scale looped results in the arxiv paper itself, so whether this composes with HRM-Text’s MagicNorm + warmup-deep-credit-assignment recipe at 1B+ is untested.
  • [2026-05-26] LT2: Linear-Time Looped Transformers: LT2 — first looped-Transformer paper to swap the inner attention primitive for subquadratic alternatives (Gated DeltaNet / DeepSeek Sparse Attention), arguing the loop synergizes with each: iterative refinement of linear-attention recurrent state, and progressive receptive-field expansion for sparse attention. LT2-hybrid (Full+GDN) surpasses the standard looped Transformer on both quality and efficiency, and Ouro-hybrid-1.4B (converted from a pretrained Ouro looped checkpoint with ~1B tokens) is reported competitive with industry 4B models at linear-time inference. Directly addresses the cluster’s open question about whether looping survives at frontier scale — by making each loop iteration cheap rather than parameter-efficient.
  • [2026-05-25] OnePiece: Bringing Context Engineering and Reasoning to Industrial Cascade Ranking System: OnePiece — first industrial-scale (billion-user Shopee personalized search) instance of iterative latent refinement. Replaces “loop count L” with “reasoning blocks K, each of M tokens” but the structural intuition is the same: refine a user representation against cached context over K KV-cached steps; block size M is the analog of loop count as the reasoning-bandwidth knob. Adds a recipe single-trajectory looped models on this page have not used: progressive per-block supervision aligned to staged user feedback (exposure → click → add-to-cart → order), distributing the multi-task signal across depth rather than across heads. Reports +1.08–1.12% GMV/UU and +2.90% ad revenue in online A/B. Distinct from the rest of this page in that the supervision-per-step is naturally staged by the deployment domain rather than designed as a curriculum.
  • [2026-05-25] The Serial Scaling Hypothesis: Position / formal-results paper providing the complexity-theoretic basis for the entire cluster. Defines “inherently serial” as outside TC, places fixed-depth Transformers, SSMs/Mamba, and fixed-backbone diffusion all inside TC⁰ (so none of them solve inherently serial problems regardless of width or step count), and lists looping / weight-shared repeating layers / RNNs / autoregressive CoT as the only model classes that escape the TC⁰ ceiling. Reframes the looped-Transformer work on this page from “a clever parameter-efficiency trick” to “the architecturally necessary response to a complexity-class limitation that no amount of width fixes.” Open question this raises directly: does the matching-via-recurrence result (Hyperloop, ELT, HRM-Text) hold because loop count puts the model into a strictly larger complexity class, or is it driven by other factors (initialization, training dynamics) that would survive even if width-scaling were a valid substitute?
  • [2026-05-24] Generative Recursive Reasoning: GRAM — a stochastic / generative branch of the RRM family from the same Sungjin Ahn lineage as HRM-Text (also Mengye Ren, Yoshua Bengio). Replaces deterministic single-trajectory latent recursion with a stochastic multi-trajectory latent-variable model trained by amortized variational inference; supports both pθ(y∣x)p_\theta(y\mid x) (conditional reasoning) and pθ(x)p_\theta(x) (unconditional generation). Inference-time scaling decomposes into recursive depth × parallel trajectory sampling — the first paper on this page to make multi-hypothesis behavior and inference-time parallel compute a first-class part of the looped-transformer story. Reports improvements over deterministic recurrent and recursive baselines on structured reasoning and multi-solution constraint satisfaction.
  • [2026-05-24] HRM-Text: Efficient Pretraining Beyond Scaling: technical paper for HRM-Text, the two-stack hierarchical-recurrence variant of looped Transformers, applied at 1B / 40B-token LLM pretraining scale. Adds two new stability mechanisms (MagicNorm + warmup deep credit assignment) that no other filed looped-LLM paper uses, and shifts the matching-via-recurrence frame from “iso-compute parameter parity” to “100–900× pretraining-token reduction at competitive benchmark scores vs 2–7B open models.” Apache-2.0 weights, full recipe, ~$1500 reproduction cost.
  • [2026-05-24] Hyperloop Transformers: combines the “middle cycle” looped layout with a single mHC-style hyper-connection per loop boundary (diagonal B, no Sinkhorn projection); matches or beats depth-matched plain Transformer and full mHC baselines at 240M/1B/2B scales with ~half the parameters and negligible throughput overhead; INT4-quantizable; LoRA-on-loops baseline is strictly dominated.
  • [2026-05-24] ELT: Elastic Looped Transformers for Visual Generation (filed earlier same day): ILSD prefix-as-student training trick makes looped visual generators any-time at inference; 4× parameter reduction at FID 2.0 on ImageNet-256 vs MaskGIT-XL; 76M ELT beats 306M MAGVIT-L on UCF-101.
  • [2026-05-24] Layer-looping scaling laws — Hayden Prairie tweet: announces predictable scaling laws for the loop dimension, with the prescription that data and loop count should be scaled together at fixed parameter budget; claims a quality match with a 2× parameter non-looped Transformer.

Open questions

Section titled “Open questions”
  • Do the iso-quality / 2×-param gains hold at frontier scale? Hyperloop tops out at 2B params on 100B tokens, ELT at 1.1B on ImageNet-scale data; no public looped model has been trained at the 30B+ / trillion-token scale where modern LLMs live. The Prairie scaling laws suggest gains should persist but the data-vs-loop trade-off in the over-trained regime needs more datapoints (Hyperloop’s Table 2 already shows the gap narrows at 400 toks/param at 240M). HRM-Text trains at 1B / 40B tokens — the opposite extreme (massively undertrained) — so the two recent LLM datapoints are at non-overlapping (N, D) regions.
  • Train-time vs inference-time loop count: is the same loop axis controlling both? ELT’s “Any-Time” inference and Prairie’s training-time scaling law agree the axis is the same in spirit, but no paper has fit both surfaces simultaneously on the same model. HRM-Text’s fixed 2×(6+1) schedule is even more rigid than ELT or Hyperloop’s — is the H_cycles × L_cycles count exploitable as an inference-time compute knob the way ELT’s L is? The Swayam-Bhanded “adapt to diffusion” framing in the ingest pointer for HRM-Text: Efficient Pretraining Beyond Scaling makes this concrete. GRAM (Generative Recursive Reasoning) offers a candidate answer in the stochastic case — sample more trajectories — but does not resolve the deterministic any-time question.
  • Stochastic-recursion vs deterministic-recursion at matched parameter / compute budget: GRAM (Generative Recursive Reasoning) reports improvements over “deterministic recurrent and recursive baselines” on structured reasoning and multi-solution constraint satisfaction, but the comparator set, scales, and whether deterministic looped + ILSD-style multi-sample inference would close the gap are all open. The deterministic any-time-inference setup (ELT) is structurally single-trajectory, so multi-hypothesis behavior is one cleanly attributable axis of GRAM’s gain.
  • Does GRAM’s amortized variational objective compose with the MagicNorm + warmup-deep-credit-assignment stability stack HRM-Text needed at 1B? Both papers share Sungjin Ahn but ship as separate codebases, and at filing time no public LLM-scale GRAM result exists.
  • Single-stack looped + hyper-connections vs two-stack hierarchical recurrence vs stochastic GRAM: which is the better LLM-scale recipe at matched parameter budget? Hyperloop Transformers, HRM-Text: Efficient Pretraining Beyond Scaling, and Generative Recursive Reasoning all sit in the same family but at very different scales / metrics / token budgets and no head-to-head ablation exists.
  • Optimal HC placement appears to be loop boundaries only, contradicting mHC’s “every layer” prescription (Hyperloop Transformers Table 5). Open whether this is specific to the looped setting (looping already enforces a notion of structured depth) or a more general statement about HC frequency.
  • Mixture-of-Recursions (Bae et al. 2025; cited as inspiration by both ELT and Hyperloop) routes a dynamic number of loops per token. Not yet filed on the wiki, but suggests a third axis — adaptive loop count — beyond static train-time L and inference-time L.
  • Quantization robustness vs. depth: Hyperloop shows INT4 quantization survives in the LLM setting; whether visual-generation ELT models retain quality after INT4 quantization is untested. HRM-Text claims 0.6 GiB at int4 from the blog announcement but the arxiv paper does not quantify quantization-induced degradation.
  • Are MagicNorm and warmup deep credit assignment specific to the two-stack hierarchical-recurrence regime, or would they also lift single-stack looped LLMs at depth × token budgets where loss spikes start to appear? No filed paper has ablated this.
  • Does looping only matter because it changes the model’s complexity class (The Serial Scaling Hypothesis §2.3), or are there other independent benefits (init quality, gradient flow, regularization through weight-sharing) that would persist even on parallel problems? No filed ablation isolates the “serial-class” contribution from these alternatives. Conflict to watch: the Serial Scaling Hypothesis claim that no fixed-depth backbone diffusion model solves inherently serial problems regardless of denoising steps (The Serial Scaling Hypothesis §5) is in direct tension with the diffusion-distillation / flow-map test-time-scaling literature (Test-time scaling of diffusions with flow maps, Diffusion Distillation) that treats step count as a usable compute axis — the looped-transformer cluster sits on the “you need real depth, not more steps” side of that argument.
  • Does the OnePiece “progressive supervision per reasoning block” recipe (OnePiece: Bringing Context Engineering and Reasoning to Industrial Cascade Ranking System) generalize beyond the recsys setting where the user-feedback chain provides natural staged signals? ILSD (ELT: Elastic Looped Transformers for Visual Generation) uses ground-truth-at-every-prefix supervision; OnePiece distributes different supervision tasks across blocks. Open whether the latter pattern works when there isn’t a naturally graded signal hierarchy (clicks → orders) to anchor it.

Papers

Section titled “Papers”

2026-05

Section titled “2026-05”

2026-04

Section titled “2026-04”

2026-03

Section titled “2026-03”

2025-11

Section titled “2025-11”

2025-10

Section titled “2025-10”

2025-09

Section titled “2025-09”

2025-07

Section titled “2025-07”