dMel: Speech Tokenization made Simple

TL;DR

dMel is a train-free speech tokenization scheme that discretizes mel-filterbank energies directly into evenly-spaced intensity bins (16 bins × 80 channels per frame), then feeds these tokens — one embedding stack per frame — into an ordinary decoder-only transformer alongside character-level text tokens. It replaces the semantic-vs-acoustic-token dichotomy (HuBERT-KM, EnCodec, SpeechTokenizer) with a single interpretable representation that a mel-filterbank vocoder can invert. A single 258M unified decoder trained with dMel achieves 4.2% WER on LibriSpeech test-clean (vs 5.8% for HuBERT-KM, 6.9% for SpeechTokenizer) for ASR, and 4.3% / 1.8% WER/CER on TTS, beating larger baselines including VOXTLM-350M and USLM.

Key claims

Discretizing mel-filterbank energies into 16 evenly-spaced bins per channel changes WER on vocoder-reconstructed audio by <0.15 (2.13 → 2.23 with ParallelWaveGAN, 2.08 → 2.11 with HifiGAN), showing quantization has essentially no impact on downstream information content [Table 1, Table 3].
Within a matched 258M decoder-only architecture, dMel achieves the lowest ASR WER (4.2 test-clean, 10.4 test-other) vs HuBERT-KM (5.8, 13.8) and SpeechTokenizer (6.9, 17.5) — showing dMel preserves semantic information better than clustered SSL tokens and residual-VQ acoustic tokens [Table 7].
For TTS the same 258M dMel model reaches 4.3% WER / 1.8% CER on LibriSpeech, vs 9.5 / 4.3 for HuBERT-KM and 11.4 / 5.9 for SpeechTokenizer in the AR-only setting, and outperforms VOXTLM-350M (which uses HuBERT-KM + OPT init) at 3.5% CER [Table 4].
Different frequency channels of dMel do not exhibit the hierarchical coarse-to-fine dependency structure of RVQ acoustic tokens, so each frame’s 80 channel indices can be predicted independently and in parallel by the transformer decoder [§2.2].
The exact vocoder matters much less than the frame rate: switching HifiGAN → ParallelWaveGAN moves WER by 0.05, but reducing frame rate 80Hz → 40Hz moves it by 0.23; a 1M-parameter vocoder at 40Hz is competitive with much larger SpeechTokenizer on WER and MOS-LQO [§3.3.1, Table 3].
Span-masking speech-token spans during training mitigates exposure bias and long-form-generation collapse by forcing the model to attend to text conditioning rather than copying local acoustic correlations [§2.3].
Joint speech-text training on the same architecture (RichASR-RichTTS single model) works but underperforms specialist single-task versions on ASR (7.6 vs 4.2 test-clean) more than on TTS (4.4 vs 4.3 WER), suggesting the joint model needs additional text-only training to learn a competitive LM [Table 8].

Method

Input speech is first converted to a standard mel-filterbank spectrogram of shape (T, 80), where T is the number of frames. Dataset-wide min/max are used to define an evenly-spaced 16-value codebook per channel; each channel value at each frame is mapped to the nearest bin index, producing a (T, 80) integer tensor — the dMel representation. To feed this into a transformer, each channel gets its own learnable embedding table (16 entries × D_channel), and the 80 channel embeddings for a frame are stacked and linearly projected into a single D-dimensional frame token, matching the character text embedding dimension. Text uses a character-level vocabulary. RoPE is applied over a global position notation shared across speech, text, and (for TTS) a speaker embedding token extracted from an off-the-shelf d-vector network. The transformer is a standard decoder-only Pre-LN model in Small/Base/Large sizes; loss is cross-entropy, computed on speech tokens for TTS and text tokens for ASR, with the 80 channels of each speech frame predicted independently and in parallel. At inference, sampled bin indices are dequantized through the codebook back to mel-filterbanks and passed through an off-the-shelf vocoder (ParallelWaveGAN or HifiGAN) to produce waveform. Span-masking is applied to speech-token spans during training as a robust-generation trick.

Results

Reconstruction (LibriSpeech test-clean, HuBERT-Large ASR-WER of vocoded audio): dMel-PWG 2.23, Mel-PWG 2.13, ground truth 2.02; MOS-LQO 4.37 vs 4.40 (Mel-PWG) and 4.19 (SpeechTokenizer 65M-encoder). Human MOS 3.07 (dMel-PWG 80Hz) vs 3.19 (Mel-PWG), 3.89 (SpeechTokenizer), 3.91 (ground truth) [Table 3].
ASR (LibriSpeech, RichASR 258M): test-clean 4.2 / test-other 10.4 with dMel; HuBERT-KM 5.8 / 13.8, SpeechTokenizer 6.9 / 17.5. VOXTLM-350M (HuBERT-KM + OPT init) gets 6.5 / 17.6 [Table 7].
TTS (LibriSpeech, RichTTS 258M): dMel 4.3 WER / 1.8 CER, HuBERT-KM 9.5 / 4.3, SpeechTokenizer AR-only 11.4 / 5.9, USLM AR+NAR 356M 6.5 WER. On LJSpeech: dMel 4.0 WER (Tacotron2 4.4, FastSpeech2 6.1, VITS 6.4); on VCTK: dMel 2.2 WER (Tacotron2 4.9, VITS 24.1) [Table 4, Table 5].
Long-form TTS: dMel maintains WER ~3.0 on >20-word sequences where FastSpeech2 degrades to 9.1 — evidence that dMel + AR decoder handles long-context generation better than NAR baselines [Table 6].
Joint speech-text: single 258M model does ASR at 7.6 test-clean and TTS at 4.4 WER, worse than specialist single-task numbers, suggesting missing text-only pretraining rather than tokenization capacity is the bottleneck [Table 8].

Why it’s interesting

dMel is the load-bearing audio-tokenization primitive under Thinking Machines Lab’s Interaction Models: A Scalable Approach to Human-AI Collaboration interaction-model architecture: TML-Interaction-Small trains audio through a light dMel embedding layer with no standalone ASR/TTS encoder, which is what makes end-to-end streaming full-duplex at 200ms micro-turns viable. It’s the audio counterpart to the pixel-first move on the vision side (Vision Banana / NEO-unify) — instead of piping speech through a Whisper-scale encoder and a separate neural codec, the dMel path is a codebook-free deterministic function on a mel spectrogram plus per-frame independent softmax. Contrasts with the semantic+acoustic multi-stage stacks in Audio-Omni: Extending Multi-modal Understanding to Versatile Audio Generation and Editing (which uses a large audio encoder + separate generation head) and the SpeechTokenizer / EnCodec residual-VQ pipeline that AudioLM-style multi-stage systems rely on; sits alongside Audio Hallucination Attacks: Probing the Reliability of Large Audio Language Models (AHA) as evidence that the standard “big audio encoder + language model” recipe has hidden costs (both perceptual latency and hallucination surface) that a train-free spectrogram tokenizer sidesteps.