Bolmo: Byteifying the Next Generation of Language Models

TL;DR

Bolmo is a family of competitive, fully-open byte-level language models (1B and 7B) produced not by training from scratch but by byteifying an existing pretrained subword LM — specifically the Olmo 3 family. A small mLSTM-based local encoder ingests raw UTF-8 bytes, a non-causal boundary predictor pools bytes into variable-length patches, and the existing Olmo 3 transformer backbone runs on the patch sequence. A two-stage conversion (subword-to-byte distillation, then end-to-end training) closes the gap to the source subword model on standard benchmarks while keeping byte-level advantages on character-sensitive tasks. Bolmo 7B reportedly outperforms prior public byte-level LLMs (BLT 7B, TFree-Hat 7B, EvaByte 6.5B) by a wide margin and approaches Olmo 3 7B itself, making byte-level LMs a practical drop-in choice for the first time.

Key claims

A pretrained subword LM (Olmo 3 1B/7B) can be converted into a byte-level LM with a short additional training run, rather than requiring full pretraining from scratch [§3.2].
The architecture is a latent-tokenizer LM: per-byte mLSTM local encoder → non-causal patch boundary predictor → variable-length patches → original Olmo 3 transformer backbone → decoder back to byte resolution [§3.1].
Bolmo 7B outperforms prior public byte-level LLMs of comparable size by an average ~16.5 absolute points on character-sensitive evaluations, while approaching the source subword Olmo 3 7B on standard LM tasks [Abstract].
Inference throughput stays competitive with subword Olmo 3 when Bolmo is trained at a higher token-compression ratio (more bytes per patch), trading some accuracy for speed — i.e. compression rate is an explicit tunable post-conversion [§Inference / Results].
A two-stage byteifying procedure works better than either stage alone: Stage 1 is subword-to-byte distillation against the original Olmo 3 logits to anchor the byte model to the subword model’s distribution; Stage 2 is end-to-end byte-level training on Dolma 3 plus targeted character-level and code data [§3.2.1, §3.2.2].
The byte model inherits the post-training ecosystem of the source subword LM (instruction-tuning recipes, RLHF data, evaluation pipelines), so byteification can be done once at the base-model level without re-doing alignment [§Post-training].
The non-causal patch boundary predictor is a design choice over BLT’s entropy-thresholded patcher: it looks at the local-encoder hidden states from both sides of a candidate boundary, which the paper argues gives better segmentation than entropy of next-byte distribution alone [§3.1.1].

Method

Bolmo is structured as a latent tokenizer LM in three stages stacked on top of an Olmo 3 backbone. (1) Each UTF-8 byte is embedded and passed through a lightweight local encoder — an mLSTM stack — that builds contextual byte-level representations at full byte resolution. (2) A non-causal boundary predictor reads pairs of adjacent byte representations and emits a per-position binary decision; consecutive bytes between boundaries are pooled into a single “patch,” producing a variable-length patch sequence whose effective compression rate (bytes per patch) is controllable at training time. (3) The patch sequence is fed into the unchanged Olmo 3 7B (or 1B) transformer backbone. An upsampler broadcasts each patch’s output back to its constituent bytes, and a small byte-level decoder produces next-byte logits.

The conversion from subword to byte happens in two stages. Stage 1 — subword-to-byte distillation: the byte model is trained to match the next-byte distribution induced by Olmo 3’s next-subword distribution under a fixed BPE tokenizer; this gives the byte model a strong initial alignment with the subword model’s behavior without re-doing the entirety of pretraining. Stage 2 — end-to-end training: the byte model is trained directly on Dolma 3 plus open code datasets and a small slice of targeted character-level data, with the boundary predictor receiving gradient through a relaxation. The Olmo 3 backbone weights are unfrozen in Stage 2; the local encoder and boundary predictor are trained from scratch in both stages.

Inference-throughput parity with subword Olmo 3 is recovered by training at a higher compression rate (forcing larger patches) — the same backbone runs on a shorter patch sequence and amortizes the byte-level overhead. The paper reports this as a knob, not a free lunch.

Results

Bolmo 7B and Bolmo 1B are released with full code (allenai/bolmo-core) and checkpoints (allenai/Bolmo-7B, allenai/Bolmo-1B) under Apache-2.0; Bolmo 1B is based on OLMo-2-0425-1B and Bolmo 7B on the Olmo 3 7B base [Abstract, model cards].
Against BLT 7B, TFree-Hat 7B, and EvaByte 6.5B, Bolmo 7B is the strongest overall byte-level model across code, math, multiple-choice QA, and character-sensitive tasks; the average improvement over the next-best prior byte LM is +16.5% absolute [Abstract, §Results].
On standard subword-friendly LM benchmarks, Bolmo 7B is comparable to (and in some cases competitive with) the source subword Olmo 3 7B — i.e. the byteification is largely capability-preserving rather than capability-degrading [§Results].
Inference speed approaches subword Olmo 3 when Bolmo is trained at higher compression rates [§Inference].

Why it’s interesting

Bolmo is the most direct practical counterexample yet to the assumption that byte-level LMs require training from scratch. It pairs naturally with two earlier filed papers from overlapping author teams: Compute Optimal Tokenization (same Limisiewicz + Zettlemoyer, same Olmo/AI2 lineage) argues from scaling laws that the right invariant is bytes-per-parameter and that BPE-derived “20 tokens/param” is a tokenizer artifact — Bolmo is the operational follow-through, removing the BPE tokenizer outright from a pretrained model. Dynamic Chunking for End-to-End Hierarchical Sequence Modeling (H-Net, Goomba Lab) is the parallel “train byte-level from scratch with learned segmentation” recipe; Bolmo’s contribution over H-Net is showing the same architectural pattern (local encoder → learned chunker → backbone) works as a cheap retrofit rather than requiring a fresh pretraining run. The Olmo-3 reuse also matters for the open-source story tracked under Open foundation-model releases: post-training (RLHF, instruct tuning, eval harnesses) can be amortized across subword and byte variants of the same base, lowering the cost of byte-level adoption for downstream labs. Finally, Bolmo connects to Karpathy: pixels may be better LLM inputs than text tokens (DeepSeek-OCR springboard) from a different direction — Karpathy argues pixels should replace text-token inputs to delete the tokenizer; Bolmo keeps text-as-input but deletes the tokenizer by going one level lower. The two are alternative escape routes from the same problem.