On the Biology of Claude's Tokenizer

TL;DR

Sander Land reverse-engineers Anthropic’s Claude tokenizer via black-box token-count probing and argues it is not the byte-level BPE that every other frontier lab ships. The reconstruction, released as the ctok library, matches Claude token counts exactly across 500+ natural languages and 22 programming languages, and pins the vocabulary at ~49-55k tokens for Claude v3-v4.6 and only ~15-20k for v4.7+ — an order of magnitude smaller than typical frontier tokenizers. The two load-bearing design choices are (1) a minimum-piece / PathPiece-style segmentation (not pairwise BPE) and (2) explicit begin-of-word ^ and end-of-word $ boundary tokens that eat inter-word spaces, plus a “CapsCode” shift-marker + lowercase pattern that removes the need for separate casing variants of the same lemma. For Luma this is a rare open datapoint on what a production alternative to standard byte-BPE looks like — and a concrete counter-recipe to the byte-level-tokenizer-free direction (Byte-Level Language Models) that argues subword tokenization is a load-bearing design decision, not an engineering detail.

Key claims

• Claude’s tokenizer is not pairwise BPE: the Korean syllable 최 is a single token but neither of its 2-byte UTF-8 prefix/suffix appears as a token in any other Hangul character sharing that prefix/suffix, violating BPE’s requirement that every non-base token decompose into two smaller vocab tokens [§Claude’s tokenizer is not BPE].
• The observed token counts are consistent with a minimum-piece tokenization in the MinGram / PathPiece family — pick the segmentation that minimizes the number of pieces given a fixed vocab — rather than a greedy pairwise merge [§Claude’s tokenizer is not BPE].
• Estimated vocabulary size is 49-55k tokens for Claude v3-v4.6 and ~16-20k for v4.7+, roughly an order of magnitude smaller than the ~100-200k vocabularies typical of open frontier tokenizers [§Vocabulary size].
• Word-like spans are wrapped in explicit begin/end boundary tokens (^word$): “semiconductromagnetusercontent” tokenizes to three tokens [^semiconduct][$][^][usercontent$][^][romagnet][$] because each morpheme is a token only in the correct positional context [§Boundary markers on words].
• Boundary markers “eat” spaces: the sequence $<space>^ is normalized away before encoding and re-inserted as a space on decode, so most inter-word spaces are effectively free and the model does not need to learn separate space-prefixed and non-space variants of the same lemma [§Boundary markers eat spaces].
• Capitalization is handled via a “CapsCode” pattern — a shift marker (↑) or caps-lock marker (⇪) followed by the lowercased form — applied per-pretoken (NASA → ⇪ nasa, Token → ↑ token), but only when the whole pretoken is title-case or all-caps; mixed forms like GaN or WiFi are stored literally [§CapsCode].
• The v4.7+ tokenizer simplifies CapsCode by dropping the caps-lock marker, which — combined with the vocabulary shrinkage — is consistent with the reported ~40% token-cost inflation for basic English text on newer Claude versions [§CapsCode, §Whitespace].
• Han characters, Korean syllables, emoji, and rare scripts do not get word boundary markers and are always tokenized character-by-character with UTF-8 byte fallback restricted to within a single character and strictly prefix-based, à la SCRIPT-BPE [§Odds and Ends].
• A fixed 7-8 token per-message overhead comes from Anthropic’s chat formatting envelope starting with ^ and ending in \n\n, not from the tokenizer’s core algorithm — which matters for anyone measuring per-request token cost [§Message overhead].

Method

Land’s approach is entirely black-box: he issues API calls to Claude and reads the reported input token count, then designs test strings that discriminate between candidate tokenizer hypotheses. The BPE-vs-minimum-piece test is a nice example — pick a Hangul syllable that Anthropic almost certainly tokenized as one piece (a common surname), then check whether any other Hangul character sharing its 2-byte UTF-8 prefix or suffix costs fewer than 3 tokens; under pairwise BPE at least one such prefix or suffix must itself be in the vocabulary, but no such case is observed, ruling out standard BPE.

The ^/$ boundary discovery uses similar differential probes: “semiconduct”, “usercontent”, and “romagnet” each cost the expected small number of tokens in isolation but the concatenation “semiconductromagnetusercontent” is only 3 tokens, which requires the tokens to be ^word or word$ forms that only fire in the correct positional context. CapsCode is detected by comparing nasa / Nasa / NASA token counts against GaN / WiFi — the former three all differ by a constant per-pretoken offset, the latter two do not. Reconstructed vocabularies are cross-checked by exact-token-count reproduction across 500+ natural languages and 22 programming languages (a corpus large enough that residual disagreements — up to 10% on Devanagari — are visible and reported as unexplained). Prior corroboration comes from token streams accidentally exposed in the sources for Anthropic’s own Biology of a Large Language Model attribution graphs paper, and from a companion project (Fable 5) that independently pinned down the three CapsCode cases.

Results

• ctok (open-sourced at github.com/sanderland/ctok) reproduces Claude token counts exactly on 500+ natural languages and 22 programming languages for both the v3-v4.6 and v4.7+ tokenizer families [§Body].
• Vocabulary estimates: 49-55k tokens (v3-v4.6) and ~15-20k tokens (v4.7+), roughly one order of magnitude below the ~100-200k typical of open frontier tokenizers like GPT-4o’s and Qwen’s [§Vocabulary size].
• Residual per-language error is 0% on most, up to ~10% on Devanagari and heavily-marker-using scripts, and generally larger on v4.7+ than on the older family [§Disclaimers].
• Practical implication: the ~40% inflation in per-token cost between old and new Claude versions on basic English text — puzzling if you assume tokenizer stability — is partly explained by the vocabulary shrinkage and the removal of CapsCode [§Body, §CapsCode].

Why it’s interesting

Almost every filed tokenizer paper on the wiki asks “should we replace subword tokenization with byte- or pixel-level input” (Dynamic Chunking for End-to-End Hierarchical Sequence Modeling H-Net, Bolmo: Byteifying the Next Generation of Language Models Bolmo, Compute Optimal Tokenization, Karpathy: pixels may be better LLM inputs than text tokens (DeepSeek-OCR springboard)). This post is the counter-datapoint: the frontier lab with the smallest total token-budget-per-answer footprint is not using byte-level or fixed BPE — they are using an explicit-boundary + minimum-piece + caps-shift design with a tiny vocabulary, and are still trimming it further across model generations. That directly informs the Byte-Level Language Models concept’s TL;DR framing that “tokenizer choice is a load-bearing modeling decision” — Anthropic evidently agrees, but has bet in a different direction than the byteification crowd. It also complements Compute Optimal Tokenization‘s finding that optimal compression rate depends on FLOP budget: Claude’s small-vocab / boundary-marker recipe pushes the effective per-word piece count around in a very different way than raising or lowering BPE’s bytes/token ratio, and would need a separate analysis to place on the same scaling curve.

See also

• Byte-Level Language Models — this is the counter-recipe: Anthropic keeps subword tokenization but redesigns what the subwords are, rather than removing tokenization altogether.
• Compute Optimal Tokenization — the closest theoretical framing; compression rate as a compute-dependent scaling axis, but assumes byte-level BPE as the underlying method.
• Dynamic Chunking for End-to-End Hierarchical Sequence Modeling — H-Net’s learned dynamic chunker is the opposite bet from CapsCode + explicit ^/$ markers; both attack the “spaces + case as first-class inputs” problem but from opposite ends.
• Bolmo: Byteifying the Next Generation of Language Models — Bolmo byteifies a pretrained subword LM; the natural inverse question raised by this post is whether Claude’s fine-vocabulary tokenizer could be byteified the same way, and whether the CapsCode / boundary-marker structure would survive that conversion.
• Karpathy: pixels may be better LLM inputs than text tokens (DeepSeek-OCR springboard) — Karpathy’s argument that pixels may be better inputs than text tokens is the maximalist position; this post shows the frontier-lab minimalist position is still investing heavily in subword tokenizer design.