Hyperloop Transformers

TL;DR

Hyperloop Transformers combine two recent architectural primitives — looped Transformers (weight-shared middle block applied recurrently) and manifold-constrained hyper-connections (mHC, matrix-valued residual streams) — into a single parameter-efficient LM architecture. The construction is minimal: organize the model into begin / middle / end blocks (the “middle cycle” recipe from prior work), loop only the middle block, and apply hyper-connections at the loop level rather than at every layer, with a simplified data-dependent diagonal (not Sinkhorn-projected) transition matrix and a learned loop-position embedding. At 240M / 1B / 2B unrolled-depth scales on FineWeb-Edu, Hyperloop matches or beats both depth-matched plain Transformer and mHC baselines while using ~50% fewer parameters, and the gains survive INT4 post-training quantization. Throughput overhead vs. a plain Transformer is negligible — much less than mHC’s specialized-kernel overhead — because the hyper-connections only fire once per loop.

Key claims

Vanilla looped Transformers underperform depth-matched non-looped baselines on perplexity (e.g. 14.85 vs 14.65 at the 240M-class depth-16 setting), confirming prior findings that pure parameter sharing across depth costs quality even when downstream task accuracy is comparable [§4.2, Table 1].
Adding hyper-connections only at the loop level (3 loops → 3 hyper-connections, applied between iterations of the middle block) outperforms every other placement tested: every-layer (12 HCs), every-2-layers, every-3-layers, every-6-layers, every-12-layers — i.e. more hyper-connections is not better in the looped setting [§4.3, Table 5].
A data-dependent diagonal transition matrix B (sigmoid over a diagonal matrix) outperforms both the identity baseline and the doubly-stochastic Sinkhorn-Knopp parameterization from mHC (PPL 14.40 vs 14.59 vs 14.61 at 136M / 12.5B tokens) [§4.3, Table 6].
At three scales the Hyperloop variant matches or beats the depth-matched plain Transformer with roughly half the parameter count: 14.40 vs 14.65 PPL at 240M-class; 9.65 vs 10.19 at 1B-class; 8.49 vs 8.60 at 2B-class [§4.2, Table 1].
Gains persist through INT4 GPTQ post-training quantization (e.g. 14.68 Hyperloop vs 14.85 Transformer at the smallest scale), addressing the concern that looped models — whose shared layers see “more” effective inputs — would be harder to quantize [§4.2 / “Post-training quantization”, Table 1].
Hyperloop adds essentially no training-throughput overhead vs the plain Transformer (e.g. 750K vs 786K toks/s at smallest scale), in contrast to a straightforward mHC implementation which drops to 514K toks/s; the loop-level placement and diagonal B together make the extra cost ~0 without needing specialized kernels [§4.2 / “Training efficiency”, Table 1].
Hyperloop continues to outperform the depth-matched non-looped baseline in an over-trained regime (20× Chinchilla-optimal, ~400 tokens/parameter at the 240M scale: 12.19 vs 12.15 PPL Hyperloop vs Transformer, while vanilla Looped slips to 12.56) [§4.2 / “Training for more tokens”, Table 2].
Increasing the number of parallel residual streams n shows diminishing returns past n=4 [§4.3, Table 4]; increasing the number of loops past 3 also shows diminishing returns at fixed parameter count [§4.3, Fig. 2].
LoRA-on-loops (relaxed Recursive Transformers — letting each loop’s middle-block weights deviate via a learned LoRA) needs rank 32 (143M params) to barely improve over vanilla Looped (14.77 vs 14.85), and still loses to Hyperloop at 136M params (14.40), so Hyperloop is a strictly more parameter-efficient way to break the per-loop weight-tying [§4.3, Table 7].
Cross-loop representation cosine similarity is lower in Hyperloop than in vanilla Looped (Table 8), supporting the authors’ hypothesis that hyper-connections give the looped block a more flexible per-iteration trajectory while still sharing the bulk of the parameters [§4.4 / “Representation similarity”].
Logit-lens analysis shows both Looped and Hyperloop residual streams are more aligned with the vocabulary distribution at each loop boundary than the non-looped baseline, suggesting Hyperloop might be especially amenable to early-exit / dynamic-depth inference [§4.4 / “Logit lens”, Fig. 4].

Method

The architecture is the “middle cycle” looped Transformer (begin block → middle block × L loops → end block) with hyper-connections applied between loops of the middle block. After the begin block produces a single residual stream h_b, the model copies it into n parallel streams to form a d×n matrix H, then enters a loop: at each iteration the input-dependent read-in A, write-out B (diagonal, sigmoid-parameterized — not Sinkhorn-projected as in mHC), and mixing get applied, the (single-stream) middle block runs, a learned loop-position embedding is added, and the n-stream representation is updated. After L loops the streams are averaged into a single residual and fed into the end block.

Three departures from the original mHC formulation: (1) the residual mixing matrix uses a simpler data-dependent diagonal (sigmoid) parameterization rather than mHC’s Sinkhorn-projected doubly-stochastic matrix — fewer parameters, less compute, and per Table 6 actually better perplexity; (2) a learnable per-loop position embedding is added after the middle block, recasting the architecture as a “depth-wise RNN” with matrix-valued hidden states and the loop embedding playing the role of an input at each “time” step; (3) hyper-connections are applied only between loops — not at every layer as in mHC — so a 3-loop model has just 3 hyper-connections regardless of how many layers are inside the middle block. The compute and parameter overhead over a plain Looped Transformer is reported as 150–300K extra parameters, and the throughput hit vs a plain Transformer is roughly 4-5% in a straightforward PyTorch + torch.compile implementation (no custom kernels).

Training uses FineWeb-Edu, Llama-2 tokenizer, AdamW with linear warmup + cosine decay, off-the-shelf hyperparameters (no tuning for the new architecture). The middle block is looped 3 times in the headline runs, with the begin/middle/end block split allocated roughly 25%/50%/25% of available parameters, yielding looped models with half as many unique parameters as their depth-matched baselines.

Results

240M class, 12.5B tokens, depth-16 unrolled: Plain Transformer 238M params, 14.65 PPL (14.85 INT4); mHC 241M, 14.55 (14.73); plain Looped [2L-4L-2L] 136M, 14.85 (15.18); Hyperloop [2L-4L-2L] 136M, 14.40 (14.68). Task accuracy 41.6% (Hyperloop) vs 41.1% (Transformer). Hyperloop achieves the best BF16 and INT4 PPL while using 43% fewer params than the Transformer [Table 1].
1B class, 50B tokens, depth-18 unrolled: Hyperloop [3L-4L-3L] reaches 9.65 BF16 PPL vs 10.19 for the Transformer baseline (~2× the params); task accuracy 49.8% vs 48.0% [Table 1].
2B class, 100B tokens, depth-38 unrolled: Hyperloop [4L-10L-4L] reaches 8.49 BF16 PPL vs 8.60 for the Transformer baseline; task accuracy 54.6% vs 52.8% [Table 1].
Over-trained (400 toks/param) 240M-class: plain Transformer 12.15, mHC 12.16, plain Looped 12.56, Hyperloop 12.19 — over-training erodes most of Hyperloop’s advantage over the dense Transformer (which catches up) but does not erode its advantage over the plain Looped baseline [Table 2].
Ablation — middle block size at fixed depth: shrinking the middle block from [3L-4L-3L] to [3L-2L-3L] reduces a 1B-class Hyperloop to 477M parameters while still outperforming the full 1B plain Transformer; further parameter savings are achievable at a small quality cost [Table 3].
Ablation — HC placement: loop-level HCs (3 total) beat layer-level (12 total), 2-layer, 3-layer, 6-layer, and 12-layer placements (14.40 vs 14.45 / 14.50 / 14.50 / 14.50 / 14.63) — more is not better, the loop boundary is the right place [Table 5].
Ablation — B parameterization: identity 14.61, Sinkhorn 14.59, diagonal (this paper) 14.40 — the simpler diagonal parameterization is both cheaper and better in this setting [Table 6].
Ablation — LoRA-Looped comparison: rank-32 LoRA on the middle block (143M params) reaches 14.77 PPL, still worse than Hyperloop at 136M (14.40); LoRA-on-loops is a strictly less parameter-efficient way to break per-loop weight-tying [Table 7].
Representation analysis: Hyperloop’s cross-loop cosine similarity is consistently lower than vanilla Looped’s at all three model scales, and Hyperloop’s residual stream shows stronger logit-lens alignment to the vocabulary at each loop boundary than the non-looped baseline [§4.4, Figs. 3–4, Table 8].

Why it’s interesting

This is the third looped-Transformer paper to land on the wiki in a week, after ELT: Elastic Looped Transformers for Visual Generation (ELT, visual generation, intra-step compute lever) and Layer-looping scaling laws — Hayden Prairie tweet (Hayden Prairie’s layer-looping scaling-law tweet, LLM pretraining). Together they triangulate the same headline finding — a looped block at iso-parameters can ~match a 2× larger non-looped Transformer — from three angles: ILSD-trained any-time visual generators (ELT), pretraining scaling laws (Prairie), and the LLM perplexity/quantization/over-training story here. Hyperloop’s specific contribution is the combination with mHC: Manifold-Constrained Hyper-Connections (mHC): adding matrix-valued residual streams at loop boundaries only, with a simpler diagonal B than mHC’s Sinkhorn-projected doubly-stochastic one. The win here is interpretive — vanilla Looped underperforms the dense baseline, mHC by itself slightly outperforms it, but combining looping with one-per-loop mHC outperforms both and is cheaper than mHC alone. This contrasts directly with mHC’s framing of the residual stream as a new scaling axis: Hyperloop suggests the lift you get from hyper-connections depends sensitively on where they are placed, and that “every layer” (mHC’s default) is not the right answer once weight-sharing across depth is in the picture.