Token Superposition Training (TST) — 2-3× wall-clock pretraining speedup without architectural change (Nous Research)

TL;DR

Nous Research announces Token Superposition Training (TST), a modification to the standard LLM pretraining loop that the authors claim delivers a 2-3× wall-clock speedup at matched FLOPs without changing the model architecture, optimizer, tokenizer, or training data. During the first third of training, the model reads and predicts contiguous bags of tokens — embeddings of multiple consecutive tokens are averaged on the input side and a modified cross-entropy predicts the next bag on the output side. For the remaining two-thirds, training reverts to standard next-token prediction. Crucially, the inference-time model is identical to one produced by conventional pretraining (no extra heads, no extra parameters, same tokenizer, same decoding loop). Validated at 270M, 600M, and 3B dense scales and at 10B-A1B MoE.

Key claims

TST is a training-time only recipe: the model architecture, optimizer, tokenizer, and training data are unchanged; the inference-time model is identical to a conventionally pretrained one [tweet body].
During the first third of pretraining, the model reads and predicts contiguous bags of tokens — input-side: averaged embeddings over the bag; output-side: a modified cross-entropy that predicts the next bag rather than the next single token [tweet body].
For the remaining two-thirds of pretraining, training reverts to standard next-token prediction [tweet body].
The reported wall-clock speedup at matched FLOPs is 2-3× over a standard next-token pretraining baseline [tweet body].
The method is validated at three dense scales (270M, 600M, 3B) and at one MoE scale (10B-A1B), per the announcement [tweet body].

Method

(All from the tweet body — no paper or code release is linked.) TST treats early pretraining as a coarser-grained sequence-modeling problem. Concretely, contiguous spans (“bags”) of tokens are summarized on the input side by averaging their token embeddings, and the supervision target on the output side is the next bag of tokens under a modified cross-entropy. The first one-third of the training run uses this bagged objective; the final two-thirds use the standard single-token autoregressive objective. The tokenizer and the model’s input/output vocabularies are unchanged, so the trained checkpoint can be deployed without any inference-time modification. The announcement does not describe bag-size selection, how the input-side averaging interacts with positional embeddings (each bag presumably occupies one position), or how the loss is normalized across bag sizes — these are open until a writeup lands.

Results

The tweet reports only:

2-3× wall-clock speedup at matched FLOPs vs. standard pretraining.
Validation at 270M, 600M, 3B dense and 10B-A1B MoE.

No FID/loss curves, downstream-eval numbers, ablations on bag size, or comparisons to MTP (multi-token prediction, DeepSeek-V3 style) or to Meta’s Better & Faster LLMs via Multi-token Prediction (Gloeckle 2024) are included in the announcement. A figure is attached to the tweet but not transcribed in the fetched body.

Why it’s interesting

This sits in the same class of “modify only the early training-loop signal, keep everything else fixed” recipes as recent published work this wiki has filed. The closest sibling is Compute Optimal Tokenization (Limisiewicz et al., Meta FAIR/UW), which makes the tokenizer’s compression rate a third scaling-law axis — TST effectively does this dynamically, by training in a high-compression regime (averaged bags) then switching to a low-compression regime (single tokens). It also rhymes with the “early-training curriculum” thread that runs through Decouple Searching from Training: Scaling Data Mixing via Model Merging for Large Language Model Pre-training (data-mixture decoupling) and the noise-curriculum levers in Diffusion training efficiency. The MTP literature (DeepSeek-V3’s multi-token-prediction auxiliary head) is the obvious comparison the announcement skips — MTP keeps the head at inference; TST drops the bagged objective entirely.

For Luma, the practical hook is that “same checkpoint, same decoder, ~2-3× cheaper training” composes cleanly with everything else: any tokenizer, any optimizer, any architecture. The risk is the absence of independent reproduction or a writeup; bag-of-tokens averaging discards order information and the announcement does not say how downstream perplexity or reasoning evals are preserved.