LT2: Linear-Time Looped Transformers

TL;DR

Looped Transformers iterate a shared block L times, but their per-token cost grows as L × N² because the loop sits on top of full softmax attention. LT2 replaces softmax with subquadratic primitives (Gated DeltaNet for linear attention, DeepSeek Sparse Attention for sparse) inside the loop, and argues the combination is synergistic in two specific ways: looping gives linear attention iterative refinement of its constant-size recurrent state, and progressively expands the effective receptive field of sparse attention. The team trains LT2-linear, LT2-sparse, and two LT2-hybrid variants (GDN+DSA, Full+GDN) under a controlled pretraining ablation, then converts a pre-trained standard looped Transformer (Ouro) into a 1.4B LT2-hybrid (Ouro-hybrid-1.4B) with ~1B tokens of conversion training, claiming it outperforms industry 1B models and is competitive with industry 4B models while keeping linear-time inference.

Key claims

Pairing looping with full softmax attention is the throughput bottleneck of looped Transformers — replacing softmax with linear or sparse attention inside the loop yields subquadratic looped models without changing the loop structure [§Abstract].
Looping interacts non-trivially with each subquadratic primitive: in LT2-linear, repeated passes act as iterative refinement of the linear-attention recurrent state (“iterative memory refinement”); in LT2-sparse, repeated passes progressively expand the effective receptive field of an otherwise local mask [§Abstract].
These two benefits are formalized theoretically and validated empirically on controlled recall, state-tracking, and language modeling tasks [§Abstract].
LT2-hybrid (GDN + DSA) — interleaving linear (Gated DeltaNet) and sparse (DSA) attention inside the loop — matches the quality of a standard looped Transformer at fully linear-time cost [§Abstract].
LT2-hybrid (Full + GDN) — interleaving GDN with a small fraction of full-attention layers — surpasses the standard looped Transformer in both quality and efficiency, identifying the same “mostly-linear with a few full-attention anchors” pattern that the broader hybrid-linear-attention literature has converged on [§Abstract].
A pre-trained standard looped Transformer can be converted to an LT2-hybrid model post-hoc; converting Ouro to Ouro-hybrid-1.4B took ~1B tokens of conversion training [§Abstract].
Ouro-hybrid-1.4B outperforms industry 1B models and is competitive with industry 4B models on the reported benchmarks, while retaining the linear-time inference speed of its underlying GDN-based block [§Abstract].

Method

LT2 is a family of looped architectures in which the inner attention primitive is one of: linear attention (Gated DeltaNet), sparse attention (DeepSeek Sparse Attention), or an interleaved mix. The loop structure itself is unchanged from prior looped Transformers — a shared block applied L times — so LT2 inherits the parameter-sharing / effective-depth properties of the looped family while removing the quadratic cost per loop iteration. The pretraining ablation systematically compares LT2-linear, LT2-sparse, and the two hybrid variants (GDN+DSA, Full+GDN) against a standard looped Transformer baseline. For the headline 1.4B release, the authors take a pre-trained Ouro looped Transformer checkpoint and replace its attention layers with the GDN+full hybrid stack, then continue training for ~1B tokens — i.e. the recipe is a conversion of an existing looped checkpoint, not a from-scratch pretrain. Code, project page, and the Ouro-hybrid-1.4B checkpoint are released (chili-lab/LT2, charlesdddd.github.io/lt2, huggingface.co/chili-lab/Ouro-hybrid-1.4B).

Results

LT2-hybrid (GDN+DSA) matches the standard looped Transformer’s quality with fully linear-time cost [§Abstract].
LT2-hybrid (Full+GDN) exceeds the standard looped Transformer on both quality and efficiency axes [§Abstract].
Ouro-hybrid-1.4B, converted from a pretrained Ouro with ~1B tokens of additional training, outperforms industry 1B baselines and is competitive with industry 4B baselines while retaining linear-time inference [§Abstract].
Quantitative numbers (perplexity, RULER, state-tracking accuracy, throughput multipliers) are reported in the paper but not extractable from the abstract-level fetch; check tables 1–3 in the PDF for exact deltas vs the Ouro baseline and the industry 1B/4B comparators.

Why it’s interesting

This is the first paper to directly fuse the Looped Transformers cluster with the Hybrid Linear Attention cluster, and the synergy claim is sharper than “stack two efficiency tricks”. The looped-Transformer literature already established that loop count is a usable axis (ELT: Elastic Looped Transformers for Visual Generation, Hyperloop Transformers, HRM-Text: Efficient Pretraining Beyond Scaling) but every prior filed paper paid the L × N² cost; the hybrid-linear-attention literature established that a few full-attention anchors restore retrieval (Kimi Linear: An Expressive, Efficient Attention Architecture, Jet-Nemotron: Efficient Language Model with Post Neural Architecture Search) but treated the architecture as a single forward pass. LT2’s claim that looping itself fixes the known weaknesses of each subquadratic primitive — linear attention’s recall via iterative state refinement, sparse attention’s locality via receptive-field growth — is a structural argument that neither parent cluster has made, and the Full+GDN result echoes the same “few full-attention anchors among many linear layers” geometry that the hybrid-linear papers converged on. Also notable that Ouro-hybrid-1.4B is a conversion from a pretrained looped checkpoint at ~1B-token cost, which makes the recipe a cheap retrofit rather than a new pretraining run — useful for the wiki’s open question about whether looped LLMs scale to frontier compute budgets.