LongCat-AudioDiT: High-Fidelity Diffusion Text-to-Speech in the Waveform Latent Space

TL;DR

LongCat-AudioDiT (paper name: LongCat-TTS) is a non-autoregressive diffusion TTS model that operates directly on a waveform latent — Wav-VAE + a diffusion backbone, no intermediate mel-spectrogram, no acoustic-token autoregression. Two ingredients beyond the architectural choice carry most of the gains: (a) replacing classifier-free guidance with adaptive projection guidance (APG), and (b) fixing a training–inference mismatch in diffusion TTS that the authors argue has gone unaddressed. The 3.5B variant beats Seed-TTS on the Seed benchmark (SIM 0.818 ZH / 0.797 ZH-Hard, up from 0.809 / 0.776) without multi-stage training or human-annotated data, and the team released both 1B and 3.5B checkpoints + code under MIT.

Key claims

Operating in waveform latent space (Wav-VAE) instead of mel-spectrograms removes the discretization stage and the second-stage vocoder, eliminating compounding errors and reducing the TTS pipeline to two components [§Abstract; §Introduction].
Replacing CFG with Adaptive Projection Guidance (APG) improves perceived naturalness and acoustic quality of the synthesized audio [§Abstract; CLI/API both expose guidance_method="apg" as the recommended setting].
The 3.5B variant achieves new SOTA voice-cloning SIM on the Seed benchmark: 0.818 on Seed-ZH (vs Seed-DiT 0.809) and 0.797 on Seed-Hard (vs Seed-ICL 0.776) [Results table, README].
Intelligibility is competitive without being SOTA — 1.09 CER on ZH (vs CosyVoice3.5 best at 0.87) and 1.50 WER on EN (vs Qwen3-TTS best at 1.23) — i.e. the design wins on speaker similarity, not on text adherence [Results table].
A long-standing training–inference mismatch in diffusion TTS is identified and rectified at the inference stage [§Abstract; not yet read in detail — the paper itself is the source for the mechanism].
Better Wav-VAE reconstruction fidelity does not monotonically translate into better overall TTS performance — the paper reports a counterintuitive interplay where stronger reconstruction can hurt downstream synthesis quality [§Abstract; ablations].
The release ships 1B and 3.5B HuggingFace-compatible checkpoints + inference code + a demo page, under MIT license, with no multi-stage training pipeline and no human-annotated data in the recipe [README §License; §Abstract].

Method

LongCat-AudioDiT collapses the standard TTS stack (text → acoustic tokens → mel → vocoder, or analogous variants) to two modules: a waveform variational autoencoder (Wav-VAE) and a diffusion backbone (the “AudioDiT” itself). At training time the diffusion model is trained to denoise in the Wav-VAE’s continuous latent space. At inference time the text encoder produces conditioning tokens; the diffusion backbone iteratively denoises a latent of duration frames (e.g. 62 latent frames for ~2.5s, with steps=16 denoising steps in the released example); the Wav-VAE decoder maps the final latent back to 24 kHz waveform. Voice cloning works by prepending the prompt audio’s encoded latent and the concatenated prompt+gen text — no separate speaker embedder.

Two inference-time changes drive the gains. First, APG replaces CFG as the guidance method (guidance_method="apg" in the API). The README does not describe APG mechanically beyond “adaptive projection guidance,” but the released code path explicitly distinguishes "cfg" and "apg" modes and the project recommends APG for voice cloning. Second, a training–inference mismatch is identified and corrected at inference — the paper itself is the source for the exact mechanism. The model is available in 1B and 3.5B sizes, both HF-compatible.

Results

On the Seed benchmark, LongCat-AudioDiT-3.5B reaches:

Seed-ZH: CER 1.09, SIM 0.818 (previous best SIM: Seed-DiT 0.809).
Seed-EN: WER 1.50, SIM 0.786 (best WER on the table: Qwen3-TTS 1.23; best EN SIM: Seed-DiT 0.790).
Seed-Hard: CER 6.04, SIM 0.797 (previous best SIM: Seed-ICL 0.776).

The 1B variant is already competitive — SIM 0.812 on Seed-ZH (above all listed prior models) and 0.787 on Seed-Hard. The narrative arc is “best speaker similarity at any size, strong intelligibility, no multi-stage training pipeline” [Results table, README].

Why it’s interesting

This is a fourth concrete instance of the Classifier-Free Guidance variants pattern: production teams are replacing CFG with a different inference-time guidance method (here, APG) rather than tuning its scalar — joining SparkVSR: Interactive Video Super-Resolution via Sparse Keyframe Propagation (Reference-Free Guidance), Random noise augmentation during diffusion training eliminates the need for CFG (noise-augmentation-as-CFG-replacement), and SoFlow: Solution Flow Models for One-Step Generative Modeling (CFG folded into the training objective). Unlike those, LongCat-AudioDiT lands on the speech side, where CFG vs APG is now a deployable knob in the released API. The “training–inference mismatch” framing also rhymes with the Ostris claim that CFG patches a noise-distribution miscalibration — worth checking whether APG and noise-augmentation are addressing the same underlying defect by different routes. As an open release, it’s another Meituan LongCat package on top of LongCat-Flash-Thinking-2601 Technical Report and LongCat-Next: Lexicalizing Modalities as Discrete Tokens — the team is now shipping coordinated open releases across reasoning LLMs, video, image, and audio.