TTS-1 Technical Report

TL;DR

Inworld TTS-1 and TTS-1-Max are two LLaMA-based autoregressive TTS models (1.6B and 8.8B inference parameters) that pair a high-resolution 48 kHz X-codec2-derived audio codec with a three-stage SpeechLM training pipeline: pre-train on 1M hours of raw audio + 20B text tokens, SFT on 200k hours of filtered audio-text pairs, then GRPO RL-alignment against a composite reward (WER + speaker similarity + DNSMOS). The codec is augmented with a transposed-conv super-resolution module that lets the same architecture emit 16/24/48 kHz audio and is trained with an explicit RMS loudness loss to keep volume stable across concatenated streaming chunks. The models support 11 languages and use inline textual audio markups (8 speaking styles, 7 non-verbal vocalizations) for fine-grained control. Code, models, and a benchmark suite are open-sourced under MIT.

Key claims

Three-stage training (audio-pretrain → SFT → GRPO) is necessary: skipping audio pretraining and SFTing directly from LLaMA-3.2-1B-Instruct gives ~15% worse WER and ~3% worse speaker similarity on the internal benchmark [§3.4].
The X-codec2 decoder is extended with interleaved strided 1D transposed convolutions + ResNet blocks to upscale acoustic-feature temporal resolution, so the same 50-tokens-per-second codebook (65,536 tokens) drives 16/24/48 kHz waveform output with minimal parameter overhead (187M → 193M params) [§2.1, Table 1].
An RMS loudness loss L_rms = MSE(log(rms(orig)+ε), log(rms(gen)+ε)) (β=10) is added to the codec’s GAN + mel + feature-matching objective to prevent perceived-volume jumps when streaming chunks are concatenated [§3.2, Eq. 3–4].
48 kHz decoder achieves DNSMOS 4.195 vs 4.110 at 16 kHz and 4.178 at 24 kHz [Table 2].
LLaMA vocabulary is expanded 128,256 → 193,856 (65,536 audio + 29 special, padded to multiple of 32); new embeddings are initialized by sampling from a multivariate normal with mean and covariance of the original embedding matrix [§2.2].
A 10% raw-text mixin (RedPajama-v2 + LAION OIG instruction data) during audio pretraining prevents text-comprehension degradation without harming speech synthesis quality [§3.3].
Mixing instruction-following text data into SFT degrades synthesis quality despite no rise in audio-token loss — text-and-audio multitask SFT is unstable here [§3.4].
GRPO is applied with reward scaling disabled (mean-centered, unscaled advantages per [Liu et al., 2025]) over a composite reward R = α·R_WER + β·R_SIM + γ·R_DNSMOS, with each component normalized to [0,1] and computed by Whisper-large-v3 + WavLM-large speaker-verification + DNSMOS on the decoded 48 kHz waveform [§3.5, Eq. 7–11].
Unified-reward GRPO outperforms three single-reward GRPO baselines on overall quality, while each single-reward run still improves its own metric vs the SFT baseline [§3.5, Fig. 6].
Style/non-verbal control is taught via LoRA fine-tuning with neutral+stylized utterance pairs from the same speaker, preserving speaker identity while adding 8 speaking-style and 7 non-verbal markup tags [§3.6, Table 5].
Training throughput: 46,000 tokens/sec/GPU (TTS-1) and 8,000 (TTS-1-Max) on 32×H100-80GB; full pre-train takes 2 and 10 days respectively [§3.3].

Method

The system has two pieces. The codec is an X-codec2 variant: a single 65,536-entry codebook merging acoustic and semantic info at 50 tokens/sec, with the decoder backbone (ResNet + Transformer → iSTFT) augmented by a super-resolution module of interleaved transposed convs and ResNet blocks. Hop length and stride configs are chosen so the same encoder feeds 16/24/48 kHz decoders by changing the upsampling factors (3,2 for 48 kHz, 1 for 16/24 kHz). Training follows X-codec2 (mel-spec loss + feature-matching + MPD+MSD adversarial loss) with the added RMS loudness term, and the 48 kHz decoder is up-trained progressively: first audio with sample rate ≥ 16 kHz, then audio with sample rate ≥ 24 kHz.

The SpeechLM is LLaMA-3.2-1B (TTS-1) or LLaMA-3.1-8B (TTS-1-Max) with the expanded vocabulary above. Pre-training is straightforward next-token prediction on 1M hours of audio (wrapped in <|speech_start|>...<|speech_end|> special tokens) + 30k hours of non-speech noise/babble + ~10% RedPajama-v2 text + LAION OIG instruction-as-text. SFT minimizes -Σ log p(audio_token_i | text, audio_<i) on 200k filtered hours, with filtering by DNSMOS bottom-20% drop, language-specific characters-per-second outlier removal (top/bottom 5%), and text-quality heuristics. SFT learning rate is initialized to the final pre-training LR — a detail the authors flag as critical for quality.

RL-alignment uses GRPO with 8 responses per prompt. The reward pipeline decodes each candidate’s audio tokens to a 48 kHz waveform, then computes WER via Whisper-large-v3 (CER for character-language entries), speaker similarity via WavLM-large speaker-verification cosine, and DNSMOS perceptual quality. Each is mapped to [0,1] (WER via exp(-α·WER), the other two by passing through their native [0,1] ranges) and linearly combined. The framework is modular and supports conditional rewards activated only on samples carrying the relevant annotation (e.g. style reward only when [whispering] is present), so emotion/non-verbal rewards can be added without contaminating neutral training samples.

Streaming inference uses context-aware decoding and concatenation at non-voicing regions to avoid audible boundaries — the RMS loudness loss in the codec is the partner trick that makes this concatenation perceptually seamless.

Results

DNSMOS by sample rate (codec only): 4.110 (16 kHz) → 4.178 (24 kHz) → 4.195 (48 kHz) [Table 2].
Pre-training ablation: audio-pretrain → SFT vs LLaMA-3.2-1B-Instruct → SFT yields ~15% lower WER and ~3% higher speaker similarity on a 5,000-utterance multilingual benchmark with 10 s mean duration [§3.4].
GRPO ablation: training each reward (WER / SIM / DNSMOS) in isolation improves that metric; a unified reward (equal weights 1/3) achieves a strictly better Pareto trade-off than any single-reward run on all three metrics vs SFT [§3.5, Fig. 6].
Throughput: 46k vs 8k tokens/sec/GPU on 32×H100s for TTS-1 vs TTS-1-Max; 2 vs 10 days of pre-training wall-clock [§3.3].
11 languages supported; the linguistic composition of pre-training is detailed in Figure 2 (heavy EN/ZH bias with smaller JA/KO/etc. tails).

Why it’s interesting

This is a clean recipe for the now-standard SpeechLM-with-neural-audio-codec TTS stack, and a useful contrast point to CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models: both use a frozen-ish ~1B LLM-style backbone over a single-codebook acoustic tokenizer with flow / vocoder downstream, both support streaming with low first-package latency, and both incorporate preference / RL alignment — but TTS-1 commits to GRPO with a composite reward and unscaled advantages, while CosyVoice 2 uses ASR-reward DPO over Gumbel-softmax token samples to avoid four-forward-pass cost. The conditional-reward design (style reward only fires on style-tagged samples) is the actually-novel piece: it sidesteps the standard multitask conflict where joint training on heterogeneous-annotation data either wastes signal or actively hurts the unannotated majority — a pattern the wiki has logged in Subliminal Learning: Language Models Transmit Behavioral Traits via Hidden Signals in Data and other narrow-finetune misalignment work.

The negative result on instruction-following SFT data (“degrades synthesis quality despite no rise in audio-token loss”) is worth filing: it’s a domain-specific instance of the broader pattern where SFT loss is not a faithful proxy for downstream quality — same flavor as Incorrect Baseline Evaluations Call Into Question Recent LLM-RL Claims arguing that loss-based comparisons mask real capability gaps in the RL-from-SFT literature. The 48 kHz codec upscaling via transposed-conv super-resolution is mechanically the same trick as the latent-image super-resolution heads in modern image diffusion, applied here so a single semantic codebook drives three sample rates from one model. Adjacent contemporaries worth tracking in the same bucket: LongCat-AudioDiT: High-Fidelity Diffusion Text-to-Speech in the Waveform Latent Space (diffusion-DiT over waveform-latent, not LM+codec), Step-Audio-EditX Technical Report (audio-LM with edit capability), Stable Audio 3 (music/SFX rather than speech), and Qwen3-TTS (version 2025-11-27) announcement — 49 voices, 10 languages, 8 Chinese dialects, ~97ms first-packet latency / Dia2 streaming TTS release — 1B/2B Apache-2.0 model with prefix conditioning / Fish Audio S2 Pro — SOTA Open-Source TTS (Dual-AR + GRPO) / Chatterbox Turbo — 350M Zero-shot TTS (Resemble AI) / Voxtral TTS — Mistral AI frontier open-weight text-to-speech model (announcement) as the parallel stream of production TTS releases the Slack channel keeps surfacing.