Dynamic Chunking for End-to-End Hierarchical Sequence Modeling

TL;DR

H-Net is a fully end-to-end, tokenizer-free language model that learns a dynamic chunking mechanism — content- and context-dependent segmentation of raw bytes — jointly with the rest of the model, replacing the BPE-tokenize-LM-detokenize pipeline with a single network. The architecture is an autoregressive U-Net: small encoder over raw bytes → learned downsampler → main network (Transformer or SSM) over compressed “chunks” → upsampler → small decoder back to byte resolution. At compute- and data-parity, one-stage byte-level H-Net beats a strong BPE Transformer baseline; iterating the hierarchy to multiple stages matches a token-Transformer of twice the size and scales better with data, with the largest gains on languages and modalities where BPE is structurally weakest.

Key claims

A one-stage byte-level H-Net outperforms a strong BPE Transformer at matched compute and data on language modeling [Abstract].
Multi-stage hierarchy further improves scaling, matching a token-based Transformer of 2× the size with significantly better data-scaling slope [Abstract].
The dynamic chunking mechanism learns content- and context-dependent segmentation jointly with the rest of the network — segmentation is not a preprocessing step but a learned module that receives gradient [§1, §2].
The H-Net is an autoregressive U-Net (Goel 2022 / Nawrot 2022 / Slagle 2024 lineage): outer encoder/decoder operate at raw resolution; inner main network operates on compressed chunks and holds the bulk of parameters [§2.1].
The main network can be any standard sequence architecture — Transformer or SSM (Mamba) — making the recipe orthogonal to the choice of backbone [§2.1].
The encoder and decoder networks face distinct design constraints from the main network because of dual objectives (representation quality + reconstruction) and the need to operate at high resolution; the paper introduces specific stabilization tricks (post-network norms, residual projections, learning-rate multipliers on the outer networks) under “H-Net (space)” → full H-Net [§2.3].
SpaceByte (a fixed-segmentation byte-level baseline) is modified into SpaceByte++ by adopting H-Net’s hierarchical structural changes (Mamba-2 outer encoder/decoder, matched layer counts) — this controlled variant isolates how much of H-Net’s gain comes from training improvements vs the chunking function itself [§Ablations / SpaceByte++].
The short causal convolution / fixed-stride patching used in prior byte/patch models (MegaByte, SpaceByte, BLT-style entropy patcher) is replaced by a learned router-style chunk boundary predictor — segmentation is data-dependent at inference time and grows with the model [§Dynamic Chunking].

Method

H-Net is structured as an autoregressive U-Net over bytes. (1) A small encoder ingests raw bytes and produces per-byte representations. (2) A dynamic chunking layer assigns each byte to a chunk via a learned boundary predictor — the predictor outputs a binary-ish decision per position that determines where chunk boundaries fall, and chunks are formed by pooling consecutive bytes’ representations between boundaries. The decision is differentiable through a relaxation so gradient flows into the predictor. (3) A main network (Transformer or Mamba-2 SSM) processes the compressed chunk sequence; this is where the bulk of parameters and FLOPs live and where the “language model” inside the H-Net does its work. (4) An upsampling layer broadcasts each chunk’s output back to the bytes it covered. (5) A small decoder at byte resolution produces the next-byte logits.

Multiple stages stack this pattern: bytes → chunks → super-chunks → main → … and back, with each level introducing its own dynamic chunker. The chunker is the key novelty over prior hierarchical byte models (MegaByte’s fixed patches, SpaceByte’s space-aware fixed segmentation, BLT’s entropy-thresholded patcher in Compute Optimal Tokenization): in H-Net the segmenter is learned end-to-end and adapts to content and context, so the same byte sequence can be segmented differently depending on what’s around it.

The training-improvements axis is reported separately: H-Net (space), an ablation that pins the chunking to whitespace boundaries but keeps H-Net’s other architectural changes (Mamba-2 outer networks, post-network norms, residual projections, outer-network LR multipliers), already improves over SpaceByte++ — so H-Net’s gains decompose into a “better hierarchical training recipe” component and a “dynamic chunking is better than fixed chunking” component.

Results

One-stage byte-level H-Net beats a Transformer++ (Mamba/Dao naming for the modernized BPE-Transformer baseline) at compute- and data-matched scale on language modeling [Abstract].
Multi-stage H-Net matches a 2× larger token-Transformer at equal compute and shows a steeper data-scaling exponent (Abstract: “significantly better scaling with data”), implying the gap grows with training scale [Abstract].
Improvements are larger on languages where BPE is known to be a poor fit — Chinese (no spaces), code, and DNA-like sequences are the headline modalities cited as showing the strongest lift, consistent with the bytes-not-tokens story in Compute Optimal Tokenization [§Experiments, per public summaries].
Ablations: SpaceByte++ already closes a meaningful share of the gap by adopting Mamba-2 outer encoder/decoder and H-Net’s training tricks; the remaining gap is attributed to dynamic chunking specifically [§Ablations].

Why it’s interesting

H-Net is the cleanest counter-example to date that a fixed tokenizer is necessary for compute-optimal LM scaling. It pairs naturally with Compute Optimal Tokenization, which argues that the Chinchilla “20 tokens/parameter” rule is a tokenizer artifact and that bytes/parameter is the real invariant — H-Net just operationalizes that by removing the tokenizer outright and learning the compression rate per-context. It is also the third Goomba Lab paper in the wiki, after Mamba-3: Improved Sequence Modeling using State Space Principles (SSM redesigned for inference-bound regime) and Raven Part 1 — Memory as a Set of Slots (memory as a set of slots with learned sparse routing); together they sketch a coherent program — replace each piece of the LM stack that was bolted on for engineering convenience (tokenizer, dense recurrent state, post-hoc retrieval layer) with a learned, end-to-end module. The dynamic-chunker is structurally a learned router over byte positions, which connects it to MoE Routing Design (router-as-architectural-primitive) and to the candidate sparse-ffn-via-embedding-tables theme: in all three, the model decides at inference time which compute to spend where. The hierarchical U-Net structure also makes it a candidate substrate for the wiki’s autoregressive video work — MALT Diffusion: Memory-Augmented Latent Transformers for Any-Length Video Generation and Raven Part 1 — Memory as a Set of Slots already separate fine- and coarse-grained processing in different ways.