Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies

TL;DR

The foundational Chinchilla / Kaplan scaling-law papers fix the tokenizer and sweep only model parameters $N$ and training tokens $D$ ; vocabulary size $V$ is a free hyperparameter that current LLMs pick largely by convention (Llama 2 uses 32K, Gemma-7B uses 256K at similar total parameter counts). This paper adds $V$ as a first-class scaling-law dimension, training models from 33M to 3B parameters on up to 500B characters, and derives the compute-optimal vocabulary size via three complementary methods (IsoFLOPs, FLOP derivative, and a parametric fit of a vocabulary-aware loss formula). All three converge on the same conclusion: larger models deserve larger vocabularies, but vocabulary parameters $N_v$ should scale strictly slower than non-vocabulary parameters $N_{nv}$ — specifically $N_v \propto C^{0.27}$ vs $N_{nv} \propto C^{0.49}$ and $D \propto C^{0.49}$ . Predicts Llama 2-70B should have had at least a 216K vocabulary (7× its actual 32K); empirically verifies at 3B scale that switching from 32K to a 43K prediction improves ARC-Challenge from 29.1 → 32.0 at matched 2.3e21 FLOPs.

Key claims

Vocabulary parameters follow a power law in compute — $N_v \propto C^{\gamma}$ — with the exponent $\gamma \approx 0.27$ strictly less than the non-vocabulary exponent $\alpha \approx 0.49$ and the data exponent $\beta \approx 0.49$ [§4.1, Fig. 5]. Practical consequence: as compute grows, allocate proportionally more to non-vocabulary and less to vocabulary, but never freeze $V$ .
Existing LLMs use insufficient vocabularies relative to their non-vocabulary parameter counts, when assessed under the compute-optimal Chinchilla-token-allocation assumption; the gap is largest for the largest models (e.g. Llama 2 70B: 32K actual vs 212K–231K predicted) [§5, Table 1, Fig. 2].
Compute-optimal vocabulary is regime-dependent: under undertraining (data-constrained), the optimum drops (e.g. 32K → 24K at 2.8e20 FLOPs), and under overtraining (data-excess, Llama-style), the optimum rises (32K → 43K at 2.3e21 FLOPs) [§5, Table 3, Fig. 7].
Language-modeling loss depends on the tokenizer’s vocabulary, making it unusable for cross-vocabulary comparison; the paper reformulates a unigram-normalized loss $\mathcal{L}_u$ that subtracts the frequency-weighted log-unigram probability, giving a vocabulary-insensitive metric [§2.2, Eq. 4].
The compression ratio of a BPE tokenizer — bytes per token — is well-approximated by a quadratic in $\log V$ : $\text{ratio}(V) \approx a(\log V)^2 + b\log V + c$ , fit across $V \in [4K, 96K]$ with high $R^2$ ; this lets the paper map the more-natural “training characters” axis $H$ back onto training tokens $D$ for compatibility with prior scaling laws [§2.2, Eq. 3, §A.9].
Three independently derived approaches — IsoFLOPs power-law fit, derivative of the FLOP-vs-loss curve w.r.t. $V$ , and parametric fit of a vocabulary-aware Chinchilla-form loss — agree on the predicted optimal $V$ across scales from 3B to 300B parameters [Table 1].
At 3B parameters and 2.3e21 FLOPs (overtraining regime), switching from 32K to the 43K predicted by Approach 3 improves ARC-Challenge 29.1 → 32.0, ARC-Easy 53.5 → 54.7, Hellaswag 53.0 → 54.1, and the 7-task average 50.3 → 51.6 [Table 2].

Method

Three approaches, applied to a joint sweep of 6 non-vocabulary parameter sizes (33M–1.13B), varying $V$ from 4K–96K, training on SlimPajama up to 500B characters:

Approach 1 (IsoFLOPs): For each of a discrete set of compute budgets, train models with different $(N_{nv}, V, D)$ that all hit the same FLOP total; take the minimum-normalized-loss point per bucket; fit $N_v \propto C^\gamma$ , $N_{nv} \propto C^\alpha$ , $D \propto C^\beta$ jointly, enforcing $\alpha = \beta$ (Chinchilla equal-scaling constraint).
Approach 2 (Derivative): Rewrite FLOPs as $C = f(N_{nv}, V, H)$ via the compression-ratio fit, then set $\partial C / \partial V = 0$ at fixed loss to solve for the optimal $V(N_{nv})$ ; scale up from a cheap small-model fit via a power-law extrapolation. Requires no additional runs beyond the small-model calibration.
Approach 3 (Parametric loss fit): Modify the Chinchilla loss form to be a function of $(N_{nv}, N_v, H)$ jointly, fit its 6 parameters on the collected runs, then numerically minimize over $V$ subject to a FLOPs budget. Uniquely can predict optimal $V$ under suboptimal $(N_{nv}, H)$ allocations (e.g. overtraining), which the other two approaches cannot.

The unigram-normalized loss $\mathcal{L}_u$ in Eq. 4 is the load-bearing methodological choice: without it, a larger $V$ mechanically inflates the standard cross-entropy loss (more classes to distribute over), and a fair cross-vocabulary comparison is impossible. The frequency-weighted normalization is empirically shown to correlate tightly with average bits-per-character (BPC), the tokenizer-agnostic compression metric [§A.5].

Verification at 3B parameters trains models under three data regimes: undertraining (FLOPs 2.8e20, data-constrained), compute-optimal (FLOPs 1.2e21), and overtraining (FLOPs 2.3e21, Llama-style), comparing conventional $V=32K$ against Approach 3’s prediction per regime.

Results

Predicted vs actual vocabularies for popular LLMs (Approach 1, Chinchilla-optimal data assumption) [Table 1]:
- 3B: 39K predicted (App1) / 43K (App2) / 37K (App3)
- 7B: 62K / 67K / 60K
- 13B: 83K / 91K / 81K
- 30B: 142K / 154K / 142K
- 70B: 212K / 231K / 218K — Llama 2 70B used 32K, i.e. 7× under-provisioned
- 130B: 237K / 258K / 248K
- 300B: 356K / 389K / 383K
3B verification at compute-optimal 1.2e21 FLOPs [Table 2]: $V=32K$ (0.10B $N_v$ , 67.3B $D$ ) → average 47.9 across 7 tasks; $V=35K$ (0.11B $N_v$ ) → 48.5 (+0.6).
3B verification at overtraining 2.3e21 FLOPs [Table 3]: $V=32K$ → 50.3 average; $V=43K$ (0.14B $N_v$ , 517.5B $D$ ) → 51.6 average — the switch from 32K → 43K is worth ~1.3 points averaged over 7 downstream tasks at matched compute.
3B verification at undertraining 2.8e20 FLOPs [Table 3]: $V=32K$ → 43.2 average; $V=24K$ → 43.9 (+0.7) — under data scarcity, smaller vocabularies win.
Fit strengths reported: IsoFLOPs power law fits $R^2$ high enough to extrapolate 3× beyond the training range; loss-formula fit yields the same optimal- $V$ shape as the two other approaches within a few percent [§4.1, §4.3].

Why it’s interesting

Sits directly upstream of Compute Optimal Tokenization, which is filed as the load-bearing modern reference on this question. Tao et al. establish the forward result: given a fixed tokenizer family (BPE, controlled by $V$ ), what $V$ should you pick? Limisiewicz et al.’s Compute Optimal Tokenization then argue the deeper point that the unit of the scaling law was tokenizer-bound in the first place — bytes, not tokens, is the compute-optimal invariant — and reproduce the Chinchilla 20-tokens-per-parameter rule only at BPE’s particular compression rate. Tao et al.’s Fig. 7 (optimal $V$ shifts down under undertraining, up under overtraining) is a direct empirical precursor to the “optimal compression rate decreases with compute” finding in Compute Optimal Tokenization §F2, restricted to the BPE-parametric setting. Complements How to Set the Learning Rate for Large-Scale Pre-training? on the Fitting-paradigm side (extrapolate a closed-form scaling law from cheap runs) — Approach 3 here is a vocabulary-aware Chinchilla-form fit. Sits alongside Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling (posted in the same Slack thread) as the two references establishing that vocabulary matters for scaling — Tao et al. treat $V$ jointly for input+output (BPE-style), while Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling argues input and output vocabularies should be decoupled.