Qwen3-TTS Family is Now Open Sourced: Voice Design, Clone, and Generation

TL;DR

Qwen3-TTS is the Qwen team’s open-weights TTS series at 0.6B and 1.7B parameters, shipped with three task heads (Base for voice clone, VoiceDesign for description-driven voice synthesis, CustomVoice for preset-timbre style control) and a separate 12Hz acoustic tokenizer. The headline capabilities are 3-second zero-shot voice cloning, natural-language voice design, 10-language multilingual coverage with several Chinese dialects, and a dual-track hybrid streaming architecture with a quoted ~97 ms first-packet latency. On the Seed-TTS benchmark the 12Hz-1.7B-Base variant reports 0.77 WER on Chinese and 1.24 WER on English, competitive with closed systems. Weights, demos, vLLM-Omni day-0 support, and a tech report (arXiv:2601.15621) are all on Hugging Face and GitHub under the Qwen organization.

Key claims

The released family spans six checkpoints across two sizes — Tokenizer-12Hz, plus 1.7B and 0.6B variants of Base / CustomVoice / VoiceDesign — all 10-language, all streaming, with instruction control on the 1.7B CustomVoice and VoiceDesign heads [§Released Models Description].
The acoustic representation is a self-developed Qwen3-TTS-Tokenizer-12Hz that performs discrete multi-codebook compression of speech while preserving paralinguistic and acoustic-environment information [§Introduction].
The architecture is described as a discrete multi-codebook LM that does full-information end-to-end speech modeling, explicitly framed against “traditional LM+DiT schemes” with their cascading errors and information bottlenecks [§Introduction].
A “Dual-Track hybrid streaming generation architecture” lets a single model serve both streaming and non-streaming generation, with end-to-end latency as low as 97 ms after a single input character [§Introduction].
Base models support 3-second voice cloning from a reference clip + transcript; an x_vector_only_mode=True switch trades quality for prompt-text-free cloning [§Voice Clone].
On the Seed-TTS test set (zero-shot, WER), Qwen3-TTS-12Hz-1.7B-Base reports 0.77 (zh) / 1.24 (en) — best English in the reported table and within ~0.06 of CosyVoice 3 on Chinese [§Evaluation, Seed-TTS table].
On the team’s TTS multilingual test set, the 12Hz-1.7B-Base reports lower WER than MiniMax and ElevenLabs on most of the 10 languages reported in the README table, with substantial leads on Chinese (0.928 vs 2.252 / 16.026) [§Evaluation, multilingual table].
vLLM-Omni provides day-0 offline inference support for Qwen3-TTS, with online serving promised later [§vLLM Usage].
A “Voice Design then Clone” workflow is documented: synthesize a short reference clip with the VoiceDesign model, then feed it to create_voice_clone_prompt on the Base model to get a reusable cloned persona [§Voice Design then Clone].

Method

The release is structured around a single discrete-token speech LM backbone with three task-specialized heads. Speech is first passed through the Qwen3-TTS-Tokenizer-12Hz, a multi-codebook acoustic tokenizer operating at 12 Hz that the Qwen team frames as the speech-representation primitive for the whole family. A separate 25 Hz variant is also reported in the evaluation tables but the 12 Hz tokenizer is what the released checkpoints are built on. Speech generation is then modeled as autoregressive prediction over those discrete codes, explicitly avoiding a downstream diffusion (“LM+DiT”) synthesizer in favor of a “lightweight non-DiT” decoder back to waveform.

Three task heads sit on top of this backbone. Base is the voice-clone / fine-tuning checkpoint — it takes a reference audio clip plus its transcript and synthesizes new content in that voice; the x_vector_only_mode flag drops the reference text in exchange for lower-fidelity cloning. CustomVoice adds style-controllable generation over 9 predefined “premium” timbres (e.g. Vivian for Mandarin, Ryan for English, Dylan for Beijing dialect, Eric for Sichuan dialect, Ono_Anna for Japanese, Sohee for Korean), each accepting a free-form instruct description (e.g. "用特别愤怒的语气说" / “Very happy.”) that conditions tone and emotion. VoiceDesign generates a voice entirely from a natural-language description with no reference audio.

Streaming is handled by what the README calls a “Dual-Track hybrid” architecture: a single set of weights supports both streaming and non-streaming inference, and the streaming path is quoted at 97 ms first-packet latency after a single input character. Deployment is via the qwen-tts PyPI package (HF Transformers under the hood, FlashAttention-2 recommended), or vLLM-Omni for offline batched inference. A Gradio web UI is shipped via qwen-tts-demo, with the Base model requiring HTTPS for browser microphone access.

Results

Headline benchmarks from the README:

Seed-TTS (zero-shot, WER ↓): Qwen3-TTS-12Hz-1.7B-Base = 0.77 zh / 1.24 en; -12Hz-0.6B-Base = 0.92 / 1.32; -25Hz-1.7B-Base = 1.10 / 1.49. Best Chinese WER in the table is CosyVoice 3 at 0.71; best English is Qwen3-TTS-12Hz-1.7B-Base at 1.24. F5-TTS, MaskGCT, Seed-TTS, Spark TTS, Llasa-8B, KALL-E, FireRedTTS-2 and MiniMax-Speech are all included as baselines [§Evaluation].
TTS multilingual (WER ↓): 12Hz-1.7B-Base reports Chinese 0.928, English 0.934, German 1.235, Italian (partial), all generally lower than MiniMax and ElevenLabs in the same table; the Chinese gap to ElevenLabs (16.026) is the largest [§Evaluation].
Streaming latency: 97 ms first-packet end-to-end, with streaming on by default in all released checkpoints [§Introduction].
Model sizes on Hugging Face: 12Hz-1.7B-Base is 4.54 GB; 12Hz-0.6B-Base is 2.52 GB (third-party measurement, not in the official blog).

External writeups confirm the practical envelope: the 1.7B-Base is reported to run comfortably on an RTX 3090, while CPU inference works but at RTF 3–5× on a high-core CPU.

Why it’s interesting

The release is the obvious open-weights counterpart to what Luma uses today for closed-API TTS (MiniMax-Speech, ElevenLabs) — same modality, similar quality envelope per the team’s own benchmarks, Apache-style license, vLLM-Omni day-0 serving. For Luma it sits as a baseline option for any pipeline that needs character voices, dubbing, or video-aligned dialogue — and as a target to clone/fine-tune from for project-specific voices.

Architecturally it’s a deliberate counter to the “LM-generates-tokens, DiT-renders-waveform” pattern that LTX-2 (LTX-2: Efficient Joint Audio-Visual Foundation Model) embodies on the joint-audio-video side: Qwen3-TTS argues that a discrete multi-codebook LM with a lightweight non-DiT decoder is enough, and they’re betting their streaming story on it (97 ms first-packet latency is hard to match with a DiT vocoder). Worth tracking as evidence in the discrete-token-LM-vs-diffusion debate for audio specifically — same debate that DiffThinker and WedLM are running on the language side.

The “Voice Design then Clone” pattern is the more interesting product-level detail: instead of asking the user to record a reference clip, you generate one with a description, then clone from your own generation. That’s the kind of UX primitive that survives any specific TTS backend.