Continuous Autoregressive Language Models

TL;DR

CALM replaces next-discrete-token prediction with next-continuous-vector prediction: a small autoencoder compresses chunks of K tokens (K=4 in headline experiments) into a single continuous latent vector that the AE decoder can reconstruct to >99.9% token-level accuracy, and a Transformer is trained to autoregressively predict the next such vector. Because each step now covers K tokens, the number of autoregressive steps drops by K× at matched sequence length. The catch is that there is no softmax over a finite vocabulary in continuous space, so the paper develops a likelihood-free toolkit: an Energy-Transformer generative head trained with the energy score (a strictly proper scoring rule), a Brier-score-based eval metric (BrierLM) that is unbiasedly estimable from samples alone, and a likelihood-free temperature-sampling algorithm. At matched compute, CALM with K=4 matches strong discrete baselines while doing 4× fewer autoregressive steps — surfacing semantic bandwidth per step as a new scaling axis distinct from N (params) and D (data).

Key claims

A lightweight context-free autoencoder maps K tokens into a single continuous latent vector with >99.9% token-level reconstruction accuracy at K=4 with latent dimension 128 [§2.1].
Reconstruction alone yields a latent space too brittle for downstream LM training; a robust latent requires (i) variational regularization with per-dimension KL clipping at 0.5 to prevent posterior collapse, (ii) dropout on the latent vector before the decoder, and (iii) random masking of input tokens to the encoder [§2.2].
Under the robust recipe, the encoder’s per-dimension posterior std converges to ~0.3 and the decoder still reconstructs at >99.9% token accuracy despite latent-perturbation noise of that magnitude — the property that makes downstream generative-model prediction noise tolerable [§2.2].
Standard MLE / cross-entropy is inapplicable in continuous space because the model has no explicit density; training uses the energy score, a strictly proper scoring rule that requires only sample access to the predictive distribution [§3.3.1, Eq. 9].
A single-step Energy-Transformer head (stack of residual MLP blocks fusing the backbone hidden state with a uniform noise vector) is chosen over diffusion or flow-matching heads specifically because iterative-sampling heads would erase the K× step-reduction speedup [§3.2, §3.3.3].
Inputs to the Transformer are kept in the discrete token space (K embeddings per step, compressed by a 2-layer MLP), not in the predicted latent space — feeding back latents as inputs degrades performance [§3.3.3].
Perplexity is not computable for CALM; the paper proposes BrierLM, the geometric mean of n-gram Brier scores estimated from two model samples per position, as a likelihood-free, strictly-proper alternative [§4.2, Eq. 14].
BrierLM is unbiasedly estimable using only model samples (no density evaluation), making it usable on any implicit generative LM — not just CALM [§4.2].
A principled likelihood-free temperature-sampling algorithm draws from the exact temperature distribution in theory and is paired with an efficient batched approximation [§5].
At matched compute, K=4 CALM achieves performance comparable to strong discrete-token baselines while running 4× fewer autoregressive steps — i.e. the performance–compute trade-off shifts on a new axis (semantic bandwidth per step) [§1, §Experiments].

Method

CALM has three trained components and one frozen one. The frozen-after-pretraining piece is a small variational autoencoder: an encoder of two position-wise FFNs flanking a flatten + linear projection compresses K=4 token embeddings into a 128-dimensional Gaussian posterior; the decoder mirrors this and reads tokens back via a tied embedding matrix and argmax. KL clipping (floor of 0.5 per dimension) prevents posterior collapse; latent-vector dropout (10%) and input-token masking enforce a smooth, redundant latent manifold so that the downstream LM’s prediction noise survives the decoder. The first trained component is a Transformer backbone that takes a sequence of compressed K-token input chunks (each step’s input is the embedding-then-MLP-compressed previous chunk) and emits a hidden state h_t. The second trained component is the Energy-Transformer head: a stack of residual MLP blocks (count = ¼ the backbone depth) that takes h_t plus a uniform random noise vector ε and produces a 128-d output latent v_t in a single forward pass. The head is trained with the energy loss — an unbiased Monte Carlo estimator of the energy score using M=4 model samples and N=400 target samples drawn from the encoder’s posterior — which is strictly proper for the predictive distribution while requiring no density evaluation. At inference, each step (i) embeds and compresses the previously emitted K-token chunk, (ii) runs one Transformer pass to get h_t, (iii) samples one v_t from the head, (iv) passes v_t through the frozen AE decoder to materialize the next K tokens. The third trained component is the BrierLM-side machinery: two-sample Brier estimators per position and a likelihood-free temperature-sampling procedure that biases the implicit predictive distribution toward sharper outcomes without ever computing a density. Code and project page: https://github.com/shaochenze/calm and https://shaochenze.github.io/blog/2025/CALM.

Results

K=4 CALM matches strong discrete baselines on the performance–compute frontier while reducing autoregressive steps by 4× [§1, §Experiments].
AE reconstruction stays above 99.9% token accuracy at K=4 with latent dim 128 even under heavy variational noise (per-dimension posterior std ≈ 0.3) [§2.2].
Energy-Transformer head accounts for only ~10% of total model parameters; ablating its size keeps language-modeling quality essentially flat — consistent with the “the backbone does the generative work; the head is a sampler” reading also seen in NextStep-1 [§3.3.3].

Why it’s interesting

CALM exposes a third scaling axis for LLMs that is orthogonal to parameter count and training tokens: semantic bandwidth per autoregressive step. The existing inference-efficiency literature on the wiki — speculative decoding (Speculative Decoding: Performance or Illusion?, Speculative Speculative Decoding, TorchSpec: Speculative Decoding Training at Scale), KV-cache reordering for non-AR decoders (WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inference), and lossy weight quantization (1-bit Bonsai 8B: End-to-End 1-bit Language Models from PrismML) — all work at the system layer and produce the same output distribution faster (or a lossy approximation of it). CALM is upstream: it changes the unit of prediction so the model fundamentally has fewer steps to run. It pairs naturally with Compute Optimal Tokenization, which already argued bytes-per-parameter (not tokens-per-parameter) is the right scaling invariant — CALM is what happens when you make the “what is a token” question continuous and push K well past what a finite vocabulary can support. The architectural recipe (small AE + frozen decoder + lightweight head over continuous latents) is the language analog of NextStep-1: Toward Autoregressive Image Generation with Continuous Tokens at Scale‘s image-generation stack, and its head-size ablation echoes NextStep-1’s “the head is a sampler, not the generative engine” finding. It contrasts cleanly with the diffusion-language-model line (Diffusion Language Models): CALM achieves parallelism per step via latent compression rather than via parallel masked decoding, and its single-step energy head is explicitly chosen to avoid the iterative-sampling tax that drove WeDLM and I-DLM toward strict-causal-attention reformulations.