Skip to content
Bud's Wiki

Byte-Level Language Models

TL;DR

Section titled “TL;DR”

Byte-level (or “tokenizer-free”) LMs replace BPE/SentencePiece-style subword tokenization with a learned mechanism that operates directly on raw UTF-8 bytes. Recent work shows the choice of tokenizer is not a fixed engineering detail but a load-bearing modeling decision: it sets the wrong scaling invariant (bytes — not tokens — should scale with parameters), it blocks character-sensitive tasks, and it can be removed without paying full pretraining cost. The wiki currently tracks three complementary recipes: a byteify-the-pretrained-subword-model retrofit (Bolmo: Byteifying the Next Generation of Language Models), an end-to-end learned dynamic chunker trained from scratch (Dynamic Chunking for End-to-End Hierarchical Sequence Modeling), and the scaling-law evidence that the BPE-derived “20 tokens/parameter” Chinchilla rule is a tokenizer artifact (Compute Optimal Tokenization).

Key claims

Section titled “Key claims”
  • Bytes — not tokens — are the right unit for the Chinchilla-style compute-optimal scaling rule; the BPE-specific “20 tokens/param” ratio reproduces only at that tokenizer’s particular ~4.57 bytes/token compression, and refitting in bytes gives a budget-invariant ratio (Compute Optimal Tokenization §F1).
  • The optimal compression rate (bytes per token/patch) is itself compute-dependent — it decreases monotonically as FLOP budget grows, so finer-grained tokens (toward byte-level) are increasingly optimal at scale (Compute Optimal Tokenization §F2).
  • A learned dynamic chunker — content- and context-dependent segmentation trained jointly with the rest of the network — beats fixed-stride patching (MegaByte) and entropy-based patching (BLT) at compute-matched scale, and multi-stage hierarchical byte models can match a 2× larger subword-Transformer (Dynamic Chunking for End-to-End Hierarchical Sequence Modeling Abstract).
  • A pretrained subword LM can be retrofitted into a byte-level LM with a short additional training run (subword-to-byte distillation + end-to-end byte training), preserving most of the source model’s capabilities while gaining byte-level advantages on character-sensitive tasks (Bolmo: Byteifying the Next Generation of Language Models §3.2).
  • Byteified models inherit the post-training ecosystem of the source subword LM (instruct/RLHF data, eval harnesses, downstream fine-tuning recipes) — byteification done once at the base-model level amortizes over downstream uses (Bolmo: Byteifying the Next Generation of Language Models §Post-training).
  • Inference throughput for byte-level LMs is recovered by training at higher compression rates (larger learned patches) — accuracy/speed becomes an explicit tunable rather than a fixed cost of the byte choice (Bolmo: Byteifying the Next Generation of Language Models §Inference).
  • Byte-level gains are largest where BPE is structurally weakest — code, math, character-level manipulation, and languages with high information density relative to English (high “parity”) whose multilingual-BPE compression rates sit far from each language’s per-language optimum (Compute Optimal Tokenization §Multilingual, Dynamic Chunking for End-to-End Hierarchical Sequence Modeling §Experiments).

Recent contributions

Section titled “Recent contributions”
  • [2026-05-25] Bolmo: Byteifying the Next Generation of Language Models: Bolmo. Byteifies pretrained Olmo 3 1B and 7B via mLSTM local encoder + non-causal boundary predictor + original transformer backbone; two-stage subword-to-byte distillation then end-to-end training; +16.5 absolute average over BLT 7B / TFree-Hat 7B / EvaByte 6.5B on character-sensitive evaluations while approaching source-subword Olmo 3 7B on standard LM tasks.
  • [2026-05-24] Dynamic Chunking for End-to-End Hierarchical Sequence Modeling: H-Net. End-to-end byte-level autoregressive U-Net with a learned dynamic chunker replacing both BPE and fixed-stride patchers; one-stage H-Net beats a Transformer++ at compute-/data-parity, and multi-stage H-Net matches a 2× larger subword-Transformer.
  • [2026-05-24] Compute Optimal Tokenization: Compute Optimal Tokenization. Trains 988 BLT models from 50M–7B sweeping compute, model size, and compression rate; refits the scaling law in bytes of training data and shows compute-optimal parameter scaling is proportional to bytes (not tokens) — the Chinchilla 20-token-per-param rule is a tokenizer artifact.

Open questions

Section titled “Open questions”
  • At what scale does byte-level fully overtake subword on standard LM benchmarks (vs. only character-sensitive ones)? Bolmo “approaches” but does not uniformly surpass Olmo 3 7B; H-Net matches a 2× larger subword Transformer, but the comparison is at <7B.
  • Which segmentation primitive is right — entropy-thresholded (BLT), learned content-dependent (H-Net), non-causal boundary predictor (Bolmo) — and how do they compare in a controlled head-to-head?
  • Does byteification break composition with Parameter-Efficient Finetuning? The byte model has different input geometry from the source subword model; whether existing LoRA / DoRA recipes need to be redesigned is open.
  • Is the “byteify a pretrained subword model” recipe a general substrate-conversion trick that could also be applied to convert byte → pixel (Karpathy: pixels may be better LLM inputs than text tokens (DeepSeek-OCR springboard)) or text → discrete-vision-token (LongCat-Next: Lexicalizing Modalities as Discrete Tokens)?
  • For multilingual settings, Compute Optimal Tokenization shows per-language optimal compression rates diverge in monolingual training but converge in joint multilingual training — what is the byte-level analogue, and does a single Bolmo-style byteified model handle the multilingual case better than per-language byteification?

Papers

Section titled “Papers”

2026-05

Section titled “2026-05”

2026-03

Section titled “2026-03”

2025-12

Section titled “2025-12”

2025-10

Section titled “2025-10”

2025-07

Section titled “2025-07”