CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models

TL;DR

CosyVoice 2 is a streaming zero-shot TTS model from Alibaba’s SpeechLab built on a Qwen2.5-0.5B backbone, with two key changes vs CosyVoice 1: vector quantization in the speech tokenizer is replaced by finite scalar quantization (FSQ), and the offline flow-matching mel decoder is replaced by a chunk-aware causal flow matching model that supports four attention masks (non-causal, full-causal, chunk-M, chunk-2M). The text-speech LM is restructured so streaming and non-streaming synthesis share one model — streaming mode just interleaves text and speech tokens at a fixed 5:15 ratio. The result is human-parity quality with ~150 ms first-package latency, 30–50% fewer pronunciation errors than v1, and lowest CER on the SEED-TTS hard set at filing time.

Key claims

FSQ with codebook size 6561 achieves 100% codebook utilization vs 23% (963/4096) for VQ in CosyVoice 1, and lowers ASR error on CommonVoice EN from 18.26% to 10.67% [Table 4].
A pretrained text LLM (Qwen2.5-0.5B) used as the text-speech LM backbone removes the need for both the text encoder and the utterance-level speaker embedding of CosyVoice 1 [§2.3].
Streaming and non-streaming inference live in one model: streaming uses an N:M=5:15 text-to-speech token interleaving with a “filling token” that signals “concat the next N text tokens”; non-streaming concatenates all text then all speech [§2.3].
The chunk-aware flow matching model is trained by uniformly sampling from {non-causal, full-causal, chunk-M, chunk-2M} masks per mini-batch, exploiting an implicit self-distillation where richer-context masks teach the leaner ones [§2.4].
First-package latency in streaming TTS is bounded by N·(T_lm + T_fm + T_voc) — i.e. one chunk’s worth of LM token generation, flow-matching mel reconstruction, and vocoder synthesis — and the report’s measured latency reaches ~150 ms [§2.5, §1].
An ASR-reward DPO variant is introduced that uses the ASR backend of the speech tokenizer on Gumbel-softmax-sampled tokens, avoiding the four-forward-pass cost of standard DPO over re-synthesized audio [§2.8, Eq. 14–15].
Training data: 200,000 hours (ZH+EN) for the speech tokenizer; ~167,000 hours across ZH (130k) / EN (30k) / JA (4.6k) / KO (2.2k) for CosyVoice 2 [Tables 2, 3].

Method

CosyVoice 2 keeps the predecessor’s design separation — semantic tokens generated by a text-speech LM, then converted to mel by a flow-matching model, then to waveform by a vocoder — but rebuilds each stage. The speech tokenizer is the encoder of SenseVoice-Large ASR with an FSQ module inserted mid-encoder: intermediate representations are projected to a low-rank D dimensions, each dimension’s value is quantized to {-K..K} via a bounded-round straight-through estimator, then re-projected to the original width before the rest of the ASR encoder; the speech token is the index of the quantized low-rank vector in the (2K+1)-ary system, at 25 Hz [§2.2, Eq. 1–2]. The text-speech LM is Qwen2.5-0.5B trained on two sequence templates jointly — non-streaming (S + all_text + T + all_speech + E) and streaming (5 text tokens : 15 speech tokens, with a “filling token” emitted when the model would otherwise predict text mid-chunk) — so a single set of weights supports both modes and four downstream scenarios (ICL non-streaming / ICL streaming / SFT non-streaming / SFT streaming) [§2.3, Fig. 2].

The flow-matching mel decoder is a causal convolutional Transformer UNet conditioned on (upsampled speech tokens, masked mel, speaker embedding, timestep), trained with an L1 OT-flow objective and a cosine-scheduler timestep at inference, with CFG strength 0.7 and NFE=10 [§2.4, Eq. 3–10]. Streaming is enabled by unfolding the 10-step flow estimation as a 10×-deeper causal network and training it under four attention masks sampled uniformly per batch (non-causal, full-causal, chunk-M for low first-chunk latency, chunk-2M for near-offline quality on later chunks), so the same model trades latency for quality at inference. Instructed generation is unified into the base model by collecting 1500 hours of instruction data covering emotion / speaking rate / dialects (Cantonese, Sichuanese, Shanghainese, etc.) / role-play, plus fine-grained markers like [laughter], [breath], <strong>...</strong>, <laughter>...</laughter> injected inline in text [§2.6, Table 1]. Multi-speaker fine-tuning (mSFT) prepends Speaker A<|endofprompt|> style tags so one fine-tuned checkpoint can serve many target speakers without timbre confusion [§2.7].

Results

SEED-TTS hard test set: CosyVoice 2 reports the lowest CER among compared models at filing time (numbers in Tables 5–7 of the full PDF; the abstract claims “human-parity” and the project page claims 30–50% pronunciation-error reduction vs v1).
Streaming first-package latency: ~150 ms, with claimed near-lossless quality vs non-streaming mode within the same model [§1, §2.5].
MOS (project-page eval): 5.53 for CosyVoice 2 vs 5.4 for CosyVoice 1 vs 5.52 for a commercial large-scale TTS reference.
FSQ vs VQ ablation: codebook utilization jumps from 23% → 100%; ASR error rate drops on all four eval splits — CommonVoice EN 18.26 → 10.67, CommonVoice CN 11.56 → 7.29, Fleurs EN 7.65 → 6.58, Fleurs CN 5.03 → 4.43 [Table 4].
Zero-shot crosslingual transfer: the tokenizer is trained only on ZH+EN (210k hours combined) yet generalizes to JA and KO synthesis — JA-test and KO-test (1000 samples each, drawn from CommonVoice) are released as a public benchmark [§3.1, §3.4].

Why it’s interesting

The chunk-aware mask schedule unifying offline and streaming flow-matching is the more transferable trick: it’s the same idea that the wiki’s Diffusion Distillation cluster keeps rediscovering for video — train the richer-mask variant alongside the leaner one and let the richer one self-distill into the leaner one — applied to attention mask scope rather than number of denoising steps. The 5:15 text-speech interleaving in the LM is a clean alternative to the duration-prediction / cross-attention machinery used by E2/F5/Seed-TTS NAR systems, and is mechanically the same trick as the streaming AR video work the wiki tracks (Autoregressive Adversarial Post-Training for Real-Time Interactive Video Generation, Self-Forcing++: Towards Minute-Scale High-Quality Video Generation) — small chunks, causal mask, KV reuse across chunks. The FSQ-over-VQ swap echoes a broader migration the wiki has been logging in image tokenizers (Image Tokenizer Needs Post-Training, UniTok: A Unified Tokenizer for Visual Generation and Understanding, End-to-End Training for Unified Tokenization and Latent Denoising) where codebook collapse is the dominant failure mode and FSQ-style scalar quantization is the standard fix. Worth contrasting with Qwen3-TTS (version 2025-11-27) announcement — 49 voices, 10 languages, 8 Chinese dialects, ~97ms first-packet latency (Qwen3-TTS, 49 voices / 10 languages / ~97 ms first-packet latency) which appears to be the production successor.