Compute Optimal Tokenization

TL;DR

Compute-optimal LLM scaling laws (Kaplan, Chinchilla) hold the tokenizer fixed and sweep only model size $N$ and token count $D$ . This paper makes the tokenizer’s compression rate — average bytes per token — a third axis, trains 988 BLT (Byte-Latent Transformer) models from 50M to 7B parameters with prescribed compression rates, and refits the scaling law. Headline finding: in compute-optimal configurations, parameter count scales proportionally to data measured in bytes, not tokens — so the familiar “20 tokens / parameter” Chinchilla heuristic is an artifact of the specific BPE compression rate (~4.57 bytes/token), and the bytes-per-parameter ratio is the actual invariant. A second finding: the optimal compression rate itself is compute-dependent — it decreases as FLOP budget grows, and the BPE default is suboptimal at most budgets. Results hold for latent (BLT) and subword tokenizers, and across multiple languages with language-specific optimal compression rates.

Key claims

Bytes — not tokens — scale proportionally with parameter count in the compute-optimal frontier; the Chinchilla 20-tokens-per-parameter rule is reproduced only at BPE’s particular ~4.57 bytes/token and breaks under any other tokenizer [Abstract, §F1].
Across compute budgets from 5e18 to 2e21 FLOPs, an optimal compression rate exists at each budget and decreases monotonically with budget — i.e. the larger the budget, the finer the tokens should be [§F2, Fig. 12 region].
The compression rate of popular multilingual tokenizers (Llama 3, Qwen 3) does not match the per-language optimum the paper derives; the gap is wider for non-Latin scripts and for languages with high “parity” (i.e. high information density relative to English) [§F3].
Findings transfer from BLT (latent) tokenization to BPE-style subword tokenizers when subword vocabulary size is varied to sweep compression [§Generalization].
Per-language optimal bytes-per-parameter correlates with the relative information density (parity) of that language — coarsely-encoded languages benefit more from extra data than from extra parameters [§Multilingual, Figs. 12–14].
In joint multilingual training, the optimal bytes-per-parameter ratio across languages converges to a shared value — the per-language divergence only appears in monolingual training [Appendix D].
The recommended scaling recipe: when porting a scaling recipe across tokenizers, match the ratio of training bytes to parameters, not training tokens to parameters [§Recommendations].

Method

The paper trains 988 BLT (Byte-Latent Transformer) models spanning 50M–7B parameters. BLT is a latent tokenizer whose patching scheme exposes the compression rate (bytes/token) as a controllable hyperparameter rather than a side-effect of a fixed BPE vocabulary, which is what lets the authors sweep compression independently from $N$ and $D$ . For each compute budget in {5e18, …, 2e21} FLOPs, they grid over $(N, D, \text{compression rate})$ and locate the isoFLOP-optimal point. They then refit Chinchilla-style power laws on the optimal frontier, parameterized in bytes of training data rather than tokens. Multilingual experiments repeat the procedure per-language and jointly across languages, with per-language compression controlled via the BLT patcher’s entropy threshold. Subword experiments vary BPE vocabulary size to sweep effective compression and check that the byte-not-token frontier survives the BLT→BPE transfer.

Results

988 trained models cover BLT 50M–7B across the (compute × compression) grid; an additional sweep of BPE vocabularies confirms the result holds outside latent tokenization [Abstract, §Setup].
For each compute budget, the isoFLOP-optimal bytes-per-parameter ratio is a single point on a curve that is empirically flatter in bytes than in tokens; refitting the law in bytes gives a budget-invariant ratio while the token ratio is not invariant [§F1].
Optimal compression rate decreases with compute — coarse tokens (high bytes/token) are optimal at small budgets, finer tokens at large budgets [§F2].
For Latin-script languages, the optimal compression rate scales nearly 1:1 with parity, and the popular multilingual tokenizers (Llama 3, Qwen 3) sit at higher compression rates than the per-language optimum, particularly for high-parity languages [§Multilingual, Figs. 13–14].

Why it’s interesting

Adds a new axis (tokenizer compression rate) to a corner of the wiki that has been very active. Hyperparameter scaling laws already tracks three other recent extensions to Chinchilla — How to Set the Learning Rate for Large-Scale Pre-training? (Fitting paradigm for LR), The Design Space of Tri-Modal Masked Diffusion Models (SDE-reparam + isonoise/isohorizon for tri-modal MDMs, $D^* \propto N^{0.476}$ ), and Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design (FFN-vs-attention FLOPs ratio as a scaling-law dimension). This paper is the cleanest demonstration so far that the unit of “data” in Chinchilla was tokenizer-bound, not physical; it complements the others by holding the optimizer and architecture fixed while sweeping what the previous papers held fixed — the lexicalization. Also feeds Parametric memory (where the implicit token budget bounds what can be memorized) and the candidate pixel-space-diffusion direction, where the analog question — what’s the right patch size / autoencoder compression for compute-optimal DiT scaling? — is wide open.