Fish Audio S2 Pro — SOTA Open-Source TTS (Dual-AR + GRPO)

TL;DR

Fish Speech is the open-source release repository for Fish Audio S2 Pro, a 4B-parameter multilingual TTS system trained on >10M hours of audio across 80+ languages. The key architectural move is a Dual-Autoregressive (Dual-AR) decoder: a 4B “slow” transformer predicts the primary semantic codebook of a 10-codebook RVQ codec along the time axis, and a 400M “fast” head predicts the remaining 9 residual codebooks at each timestep. Post-training uses GRPO with multi-dimensional reward signals (semantic accuracy, instruction adherence, acoustic preference, timbre similarity), and inference rides standard LLM-serving infrastructure (SGLang continuous batching, paged KV, RadixAttention) to hit RTF 0.195 and TTFA ~100 ms on a single H200. The system supports inline natural-language style tags ([whisper], [angry], plus 15,000+ free-form descriptions) for sub-word prosody/emotion control.

Key claims

Slow/Fast Dual-AR split: a 4B decoder-only transformer along the time axis emits the primary semantic codebook of a ~21 Hz, 10-codebook RVQ codec; a 400M head emits the other 9 residual codebooks at each timestep [README §Innovative Dual-Autoregressive Architecture].
Seed-TTS Eval — WER 0.54% (Chinese) / 0.99% (English), reported as best among compared closed- and open-source systems including Qwen3-TTS (0.77/1.24), MiniMax Speech-02 (0.99/1.90), Seed-TTS (1.12/2.25) [README §Benchmark Results].
Audio Turing Test (with instruction): 0.515 posterior mean, vs Seed-TTS 0.417 and MiniMax-Speech 0.387 [README §Benchmark Results].
EmergentTTS-Eval win rate 81.88% overall, with 91.61% on paralinguistics, 84.41% on questions, 83.39% on syntactic complexity [README §Benchmark Results].
Post-training uses GRPO with reward models drawn from the same data-cleaning/annotation model suite, scoring semantic accuracy, instruction adherence, acoustic preference, and timbre similarity [README §Reinforcement Learning Alignment].
Inference on a single H200: RTF 0.195, TTFA ~100 ms, throughput 3,000+ acoustic tokens/s while keeping RTF < 0.5; achieved by reusing standard LLM serving features (continuous batching, paged KV cache, CUDA Graph, RadixAttention prefix caching) since Dual-AR is structurally isomorphic to a stack of decoder-only LMs [README §Extreme Streaming Performance].
Inline style control via 15,000+ free-form natural-language tags (e.g. [whisper in small voice], [professional broadcast tone]) plus a fixed emotion library; tags can be placed at any sub-word position [README §Fine-Grained Inline Control].
Native multi-speaker generation via <|speaker:i|> tokens — one reference clip containing multiple speakers replaces the prior practice of per-speaker reference uploads [README §Native Multi-Speaker Generation].
80+ language coverage without phonemes or language-specific preprocessing; best WER on 11/24 and best SIM on 17/24 languages of the MiniMax multilingual testset [README §Robust Multilingual Support, §Benchmark Results].

Method

The system is a master-slave Dual-AR built on top of an RVQ audio codec with 10 codebooks at ~21 Hz. The “slow” AR is a 4B decoder-only transformer that runs along the time axis and predicts the primary (first) codebook — i.e. semantic content. At each emitted timestep, the “fast” AR (400M params) takes the slow AR’s hidden state and autoregresses through the remaining 9 residual codebooks before the slow AR advances. The asymmetric parameter allocation matches the asymmetric information content: the semantic codebook needs language modeling capacity, the residual codebooks need only fine acoustic detail.

Post-training is GRPO. Crucially, the same model family used for data cleaning and annotation upstream is reused as the reward model — this is presented as a fix for the usual distribution mismatch between pretraining data and post-training reward signal. Rewards are multi-dimensional: semantic accuracy, instruction adherence, acoustic preference scoring, and timbre similarity.

Because the slow AR is a plain decoder-only transformer, serving is unmodified LLM-serving (SGLang’s continuous batching, paged KV cache, CUDA Graph capture, RadixAttention prefix caching). Voice cloning uses 10-30 s of reference audio without fine-tuning. Multi-speaker conditioning is done via <|speaker:i|> control tokens learned over the reference clip’s speaker features. The repo references two technical reports: arXiv 2411.01156 (Fish-Speech v1.4, 2024) and arXiv 2603.08823 (Fish Audio S2 Technical Report, 2026).

Results

Seed-TTS Eval WER: 0.54% ZH / 0.99% EN, best among Qwen3-TTS, MiniMax Speech-02, Seed-TTS [README §Benchmark Results].
Audio Turing Test (with instruction): 0.515 — +24% over Seed-TTS (0.417), +33% over MiniMax-Speech (0.387) [README §Benchmark Results].
EmergentTTS-Eval: 81.88% overall win rate, highest among compared systems [README §Benchmark Results].
Fish Instruction Benchmark: TAR 93.3%, quality 4.51/5.0 [README §Benchmark Results].
MiniMax multilingual testset: best WER on 11/24 languages, best SIM on 17/24 [README §Benchmark Results].
Inference (H200): RTF 0.195, TTFA ~100 ms, >3000 acoustic tok/s [README §Extreme Streaming Performance].

Numbers are README-reported and use the system’s own reward model suite as part of the eval pipeline — read accordingly.

Why it’s interesting

S2 Pro is the most direct counterpart on the wiki to CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models (Alibaba CosyVoice 2): both build a streaming zero-shot TTS on top of an LLM-style autoregressive backbone with a learned quantized speech tokenization, but S2 Pro replaces CosyVoice 2’s separation (text→speech LM, then flow-matching mel decoder, then vocoder) with a single Dual-AR that emits the codec directly, and adds a GRPO post-training stage on top — putting it in the Reasoning RL lineage applied to TTS rather than to math/code. The TTFA ~100 ms / RTF 0.195 numbers continue the LLM Inference Efficiency thesis from Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decoding and the SGLang ecosystem that LLM-grade serving infrastructure transfers cleanly to other autoregressive generative domains — here the win is that decoder-only TTS gets continuous batching and prefix caching for free. The 15,000+-tag inline style interface is a sibling to Gemini 3.1 Flash TTS: Google text-to-speech with scene direction and audio tags (announcement) (Gemini 3.1 Flash TTS scene tags) and Qwen3-TTS (version 2025-11-27) announcement — 49 voices, 10 languages, 8 Chinese dialects, ~97ms first-packet latency (Qwen3-TTS 49 voices / 10 languages), pointing at convergence on natural-language control as the TTS UI rather than per-attribute sliders.