End-to-End Test-Time Training for Long Context

TL;DR

Reframes long-context language modeling as a continual-learning problem on top of a standard sliding-window Transformer rather than as a new architecture (Mamba 2, Gated DeltaNet, etc.). The model continues training on the context at test time via plain next-token prediction, compressing what it reads into its own weights; the test-time initialization is itself meta-learned end-to-end at training time, so the outer loop directly optimizes the post-TTT loss. At 3B / 164B tokens, this “TTT-E2E” recipe matches a full-attention Transformer’s loss vs. context-length curve — something Mamba 2 and Gated DeltaNet do not — while retaining RNN-style constant inference latency (2.7× faster than full attention at 128K). The paper’s central claim is that the architecture-vs-continual-learning axis is the right one for long context, not the linear-attention zoo.

Key claims

Long-context language modeling can be cast as continual learning on a standard architecture (Transformer + sliding-window attention), with the long-context capacity arising from test-time next-token-prediction updates that compress context into weights rather than from a new sequence operator [Abstract, §1].
The outer-loop training objective directly optimizes the final loss after TTT (end-to-end through the inner loop), in contrast to dynamic-evaluation-style methods that only train the initialization conventionally and run TTT only at test time [Fig. 1 caption].
TTT-E2E (this paper) is taxonomically distinct from TTT-KVB (the key–value-binding inner-loop loss used in earlier TTT-as-sequence-layer work): the inner loss here is the actual task loss (next-token CE) rather than a self-supervised KV-association proxy [§1; reference taxonomy: arXiv:2602.21204 §“Background”].
At 3B parameters / 164B training tokens, TTT-E2E scales with context length the same way as full attention on test loss, while Mamba 2 and Gated DeltaNet do not at matched compute [Abstract, Fig. 1 left].
Inference latency is constant in context length (RNN-style), giving a 2.7× wall-clock speedup over full attention at 128K context [Abstract, Fig. 1 right].
Figure 1 left: TTT-E2E “turns the worst line (green) into the best (blue) at 128K” — i.e. the same base Transformer goes from being the worst long-context loss curve without TTT-E2E to the best with it [Fig. 1 caption].

Method

The model is a standard decoder-only Transformer with sliding-window attention as its self-attention kernel — no new sequence operator, no linear-attention substitute. The new ingredient is that at test time, the model performs additional gradient steps on its own weights using next-token prediction on the given context, so the “long-term memory” lives in the weight updates rather than in a hidden state or full KV cache. Sliding-window attention provides the short-range exact-recall channel; the TTT updates provide the long-range compressed channel.

The matching training-time ingredient is meta-learning: the outer loop differentiates through the inner-loop TTT updates so that the meta-initialization is explicitly trained to be a good starting point for context-driven fine-tuning at inference. This is the “E2E” in TTT-E2E — both halves (train and test) are next-token prediction, glued through inner-loop backprop. Contrast with TTT-KVB-style sequence layers (TTT-Linear / TTT-MLP / Titans / RWKV-7), where the inner-loop loss is a self-supervised key-value association rather than the downstream task loss, and contrast with dynamic evaluation, where the initialization is trained without seeing the test-time TTT trajectory [§1, Fig. 1 caption].

Results

The headline scaling experiment is at 3B parameters trained on 164B tokens, comparing TTT-E2E against full-attention Transformer, Mamba 2, and Gated DeltaNet across context lengths up to 128K. TTT-E2E tracks the full-attention loss curve as context grows, while Mamba 2 and Gated DeltaNet fall off — i.e. the long-context degradation of linear-attention recurrent architectures isn’t intrinsic to “weights as memory”; it is intrinsic to those particular recurrent updates [Abstract, Fig. 1 left]. On latency, TTT-E2E is constant in context length and reaches a 2.7× speedup over full attention at 128K [Abstract, Fig. 1 right]. The combination — full-attention quality scaling with RNN-style latency scaling — is the headline operating point.

Why it’s interesting

For Luma, this is the cleanest argument so far that the long-context fight isn’t between attention variants but between static architectures and continual-learning recipes. If a small-context Transformer with sliding-window attention plus a meta-learned TTT loop matches full-attention scaling at 3B/164B tokens, then the right thing to instrument for long video conditioning, long-rollout autoregressive generation, and long planning traces is the outer-loop-through-inner-loop training pipeline — not yet another linear-attention substitute. The result also directly extends the team’s “test-time training as a fifth axis” reading: prior TTT-as-sequence-layer work (TTT-KVB, Titans, RWKV-7) optimized a proxy KV-association loss in the inner loop and trained the outer loop without it; this paper is the first to bind both loops to the same next-token loss at production scale.