Skip to content
Bud's Wiki

Full-bandwidth transformer

TL;DR

Section titled “TL;DR”

Autoregressive transformers compute along two axes — horizontally across generated tokens and vertically through depth — but the vertical feedback channel between decoding steps is narrow: only the sampled token returns to the bottom of the stack while the top-layer hidden state is discarded. The full-bandwidth transformer widens this channel with latent feedback: at each decoding step, the previous top-layer hidden state is fused with the sampled token’s embedding through a gated linear unit and fed back as the next input, giving non-verbalized computation a renewed depth budget on every step while preserving the standard architecture, KV cache, and LM objective. Trained with a scheduled multi-pass objective that introduces latent feedback late in pretraining, 1B-parameter full-bandwidth transformers trained on up to 400B tokens improve validation loss, 5-shot LM eval, math/coding generation, and instruction-tuned performance — matching or approaching standard transformers trained with ~1.5× more tokens, at negligible per-token decoding overhead, and producing shorter reasoning traces at equal or better accuracy.

Key claims

Section titled “Key claims”
  • The vertical feedback channel from one decoding step to the next in a standard autoregressive transformer is narrow by design: only the sampled discrete token returns to the bottom of the stack, while the top-layer hidden state is discarded [§Abstract].
  • Latent feedback fuses the previous top-layer hidden state with the current sampled token embedding through a gated linear unit and feeds the result as the next-step input, letting non-verbalized computation re-enter the stack with a renewed depth budget [§Abstract].
  • The intervention is architecture-preserving: standard transformer block, unchanged KV cache, and unchanged language-modeling objective [§Abstract].
  • A scheduled multi-pass training objective is required to train the model without losing parallel teacher forcing — latent feedback is introduced late in pretraining, and a small fraction of deeper feedback passes is mixed in for stability [§Abstract].
  • At 1B parameters trained on up to 400B tokens, latent feedback improves validation loss, 5-shot language-model evaluation, math and coding generation, and instruction-tuned performance [§Abstract].
  • Full-bandwidth transformers match or approach standard transformers trained with roughly 1.5× more tokens, at negligible per-token decoding overhead [§Abstract].
  • Full-bandwidth transformers produce shorter reasoning traces at equal or better accuracy, meaning the model spends fewer verbalized tokens per solved instance [§Abstract].

Method

Section titled “Method”

The architecture keeps a standard decoder-only transformer with its usual KV cache and next-token LM loss. The change is a single per-step fusion module at the model input: at decoding step t+1t+1, instead of embedding only the sampled token xt+1x_{t+1}, the model fuses embed(xt+1)\text{embed}(x_{t+1}) with the previous step’s top-layer hidden state hth_t through a gated linear unit, and feeds the resulting vector as the input to layer 0. Because hth_t carried the full pre-sampling computation, non-verbalized reasoning (whatever the top-layer hidden state was computing that did not end up as the emitted token) re-enters the stack at the bottom and gets a full depth budget of L more layers to run further.

Training with teacher-forced parallelism is the delicate piece: latent feedback couples adjacent decoding steps, breaking the “each position depends only on its prefix of true tokens” property that gives the transformer its parallel training. The paper’s answer is a scheduled multi-pass objective introduced late in pretraining, augmented with occasional deeper feedback passes for stability. The Abstract does not enumerate the schedule details.

Results

Section titled “Results”

Trained at 1B parameters up to 400B tokens, the full-bandwidth transformer improves over the equivalently-scaled standard-transformer baseline on validation loss, 5-shot LM eval, math and coding generation, and instruction-tuned performance [§Abstract]. The headline compute-efficiency claim is that it matches or approaches a standard transformer trained with roughly 1.5× more tokens [§Abstract] — i.e. the feedback channel gives ~1.5× data-efficiency at 1B/400B without changing the parameter count or the per-token FLOPs meaningfully. Per-token decoding overhead is reported as negligible [§Abstract]. On reasoning, the model produces shorter chains at equal or better accuracy [§Abstract], though the Abstract does not quantify the compression ratio or list which reasoning benchmarks this is measured on.

Why it’s interesting

Section titled “Why it’s interesting”

Full-bandwidth transformer sits at a specific position in the Looped Transformers design space: it recycles computation not within a decoding step (as in ELT / Hyperloop / HRM-Text, which apply a shared block iteratively at fixed depth) but across decoding steps, using the top-layer hidden state as a low-bandwidth carrier of pre-sampling computation that would otherwise be discarded at the discrete-token bottleneck. This is a different structural bet than ELT: Elastic Looped Transformers for Visual Generation (weight-shared inner block) and Hyperloop Transformers (hyper-connections at loop boundaries), but shares the same functional goal — expanding effective depth without adding unique parameters — and is directly aligned with The Serial Scaling Hypothesis‘s argument that fixed-depth models cannot solve inherently serial problems regardless of width. The “shorter reasoning traces at equal or better accuracy” result also contrasts productively with pure-text CoT: rather than emitting more tokens for extra compute, the model absorbs that compute into the latent feedback channel, an efficiency lever complementary to Inference-Time Scaling‘s emit-more-tokens paradigm.

See also

Section titled “See also”