Qwen3.5-Omni Technical Report

TL;DR

Formal technical report for Qwen3.5-Omni, the omni-modal extension of the Qwen3.5 family that processes text, image, audio, and video inputs and emits interleaved text + streaming speech outputs. Compared to the launch blog (Qwen3.5-Omni: A Native Omni-Modal Model with Thinker-Talker MoE Architecture), the report fixes five architectural specifics: both Thinker and Talker use Hybrid-Attention MoE (with a Gated Delta Net module from the Qwen3.5 line for long-context KV-cache compression), the audio encoder AuT is trained from scratch on 40M hours of Qwen3-ASR-labeled audio-text pairs, the Talker uses an RVQ multi-codebook representation with an MTP head and a streaming ConvNet code2wav, ARIA replaces the dual-track text/speech generation of Qwen3-Omni with a single monotonically-interleaved stream whose cumulative speech-to-text ratio is bounded by the item-level global ratio, and explicit text-string timestamps replace TM-RoPE’s absolute-time position IDs to keep long video positional encodings dense. Plus-variant claims SOTA on 215 audio and audio-visual subtasks, surpassing Gemini 3.1 Pro on most audio tasks and matching it on audio-visual; reported first-packet latency 235 ms (Flash audio) → 651 ms (Plus video). Trained in three pretraining stages over ~4T tokens for the General stage, then extended to 262K-token long-context.

Key claims

Architecture continues the Thinker-Talker split from Qwen2.5-Omni but moves both halves to Hybrid-Attention MoE, adding Gated Delta Net for KV-cache reduction on long audio-video sequences [§2.1, §2.5].
Audio encoder AuT is trained from scratch (not Whisper-derived) on 40M hours of audio-text pairs labeled by Qwen3-ASR; 128-channel mel input, 4× Conv2D downsampling (16×), self-attention trunk emitting tokens at 6.25 Hz; multilingual data ratio Chinese : English : other = 3.5 : 3.5 : 3 [§2.2].
Vision encoder is reused from Qwen3.5 (SigLIP2-based) and operates with dynamic frame-rate video sampling to preserve information while staying aligned with the audio stream [§2.3].
Text-string timestamps prepended to each video/audio-video temporal patch replace TM-RoPE absolute-time IDs, avoiding sparse positional encoding for long videos; audio also receives randomly-spaced text-string timestamps for cross-modal alignment [§2.3].
ARIA (Adaptive Rate Interleave Alignment) unifies dual-channel text+speech generation into a single interleaved stream under a monotonic constraint — cumulative speech-to-text token ratio at any prefix is bounded by the item-level global ratio; this replaces Qwen3-Omni’s MFA-derived or fixed-rate interleaving [§2.4, §2.5].
Talker emits RVQ codec tokens predicted by a lightweight MTP module, then converts multi-codebook tokens to waveform with a causal ConvNet codec decoder (code2wav) [§2.4, §2.5].
Talker conditions on Thinker’s history text tokens, multimodal representations, and current-turn streamed text; dedicated Talker system prompt enables zero-shot voice cloning and controllable speech generation [§2.4].
Pretraining is three-stage: S1 Encoder Alignment (LLM frozen, train vision + audio encoders separately), S2 General (all params unfrozen, 4T tokens — text 0.92T, audio 1.99T, image 0.95T, video 0.14T, video-audio 0.29T, seqlen 32,768), S3 Long Context (seqlen 262,144 with higher long-audio/video proportion) [§3].
Thinker post-training is three-stage: Specialist Distillation from domain-specialized teacher models, On-Policy Distillation that uses text-conditioned responses as targets for matched audio-conditioned queries, and Interaction-Aligned RL on multi-turn trajectories to address code-switching / persona drift / instruction-following decay [§4.1].
Talker post-training is four-stage starting from 20M hours of multilingual speech with multimodal context, then quality-stratified CPT with Qwen3-Omni-Captioner augmentation extending context to 64K, then RL alignment, then a final stage (truncated in the fetched abstract) [§4.2].
Plus achieves SOTA on 215 audio and audio-visual understanding / reasoning / interaction subtasks; surpasses Gemini 3.1 Pro on general audio understanding, reasoning, recognition, translation, and dialogue; matches it on overall audio-visual understanding [Abstract, §1].
Reported first-packet latency at 1× concurrency on internal vLLM with torch.compile + CUDA Graph for MTP and codec decoder: 235 ms (Flash audio) / 426 ms (Flash video) / 435 ms (Plus audio) / 651 ms (Plus video) [Table 2].
Linguistic coverage: speech recognition 113 languages and dialects (74 languages + 39 Chinese dialects), speech synthesis 36 (29 languages + 7 Chinese dialects), text 201 [Table 3].
The byte-level BPE vocabulary increases from 150K (Qwen3-Omni) to 250K (Qwen3.5), reported to improve encoding/decoding efficiency by 10–60% across most languages [§2.3].
Emergent capability: “Audio-Visual Vibe Coding” — directly generating executable code from audio-visual instructions, observed only in the omni-modal model [Abstract, §1].

Method

The model is built around the Thinker-Talker split from Qwen2.5-Omni but rebuilds nearly every component. The Thinker is a Hybrid-Attention MoE decoder over a concatenated sequence of text tokens (Qwen3.5 250K BPE), audio tokens (AuT output at 6.25 Hz over 16 kHz mel input), and visual tokens (Qwen3.5 vision encoder over dynamically-sampled video frames). Each video patch and each random subset of audio frames carry an explicit text-string timestamp in seconds, which replaces TM-RoPE’s absolute-time positional IDs to keep long-context positional encoding dense.

The Talker is a separate Hybrid-Attention MoE that conditions on Thinker hidden states plus the historical and current-turn text, and emits speech as RVQ codec tokens. An MTP module predicts the residual codebooks for fine-grained acoustic control; a causal streaming ConvNet (code2wav) converts the multi-codebook tokens to waveform. ARIA reformulates the dual-track generation of Qwen3-Omni as a single interleaved stream of text and speech tokens, subject to the constraint that for every prefix the cumulative speech-to-text ratio cannot exceed the global item-level ratio — this removes MFA-derived alignment and fixed-rate interleaving while still tolerating low-encoding-efficiency languages.

Streaming-time optimizations: chunked prefilling lets the audio and vision encoders emit chunks along the temporal dimension to reduce TTFT; the Hybrid-MoE backbone with Gated Delta Net cuts KV-cache I/O for long audio-video sequences; the lightweight MTP and the causal ConvNet codec decoder enable batched, low-latency speech generation. The Talker is also given a dedicated system prompt encoding target voice characteristics, which subsumes speaker embeddings and supports zero-shot voice cloning plus text+codec multimodal voice description.

Pretraining runs in three stages: S1 freezes the LLM and trains the two encoders against image-text and audio-text pairs (with adapter-only warm-up); S2 unfreezes all parameters and trains over ~4T tokens of mixed-modality data at seqlen 32,768; S3 extends to seqlen 262,144 and rebalances toward long audio / long video. Thinker post-training distills domain-specialist Qwen-3.5 SFT/RL teachers into a single model, then on-policy-distills text-conditioned responses as targets for the audio-conditioned setting to close the audio-vs-text response-quality gap, then runs Interaction-Aligned RL on multi-turn trajectories rewarding stable persona / no code-switching / instruction-following over long contexts. Talker post-training adds a multilingual general pretraining stage (>20M hours), a Qwen3-Omni-Captioner-augmented long-context CPT, and a final RL alignment stage (the fetched body truncated at this point).

Results

Audio benchmarks (Plus, self-reported). SOTA on 215 audio and audio-visual subtasks; surpasses Gemini 3.1 Pro on general audio understanding, reasoning, recognition, translation, and dialogue.
Audio-visual understanding (Plus). Reported “on par with” Gemini 3.1 Pro on overall audio-visual understanding.
First-packet latency (1× concurrency, internal vLLM + torch.compile + CUDA Graph). Flash: 235 ms audio / 426 ms video. Plus: 435 ms audio / 651 ms video. Latency stays stable as concurrency increases (235 → 352 ms on Flash audio at 8× concurrency; 435 → 955 ms on Plus audio at 8× concurrency) [Table 2].
Steady-state throughput (Tokens/s at 1× concurrency). Flash Thinker 177/171 TPS, Talker 70/70 TPS; Plus Thinker 57/54 TPS, Talker 67/67 TPS. Generation RTF 0.178 (Flash) / 0.187 (Plus) at 1× concurrency, climbing to ~0.26 / 0.33 at 8× [Table 2].
Context window. 256K tokens supporting >10 hours of audio or ~400 s of 720p video at 1 FPS — same headline as the launch blog [§Architecture].
Linguistic coverage. Speech input 113 (74 languages + 39 Chinese dialects); speech output 36 (29 languages + 7 dialects); text 201 [Table 3].
Tokenizer efficiency. Vocabulary 150K → 250K (byte-level BPE); encoding/decoding 10–60% more efficient across most languages [§2.3].
Pretraining mix (S2 General, ~4T tokens). Text 0.92T / Audio 1.99T / Image 0.95T / Video 0.14T / Video-Audio 0.29T [§3].
AuT pretraining. 40M hours of Qwen3-ASR-labeled audio-text pairs, ratio Chinese : English : multilingual = 3.5 : 3.5 : 3 [§2.2].
Talker general-stage data. >20M hours of multilingual speech paired with multimodal context [§4.2].
Emergent capability. Audio-Visual Vibe Coding — direct code generation from audio-visual instructions, reported as an emergent property of the omni-modal training [Abstract, §1].

Why it’s interesting

This is the formal technical report for the model whose launch blog post is Qwen3.5-Omni: A Native Omni-Modal Model with Thinker-Talker MoE Architecture — same architecture story, but the report locks down specifics that the blog left to PR copy: AuT’s 40M-hour pretraining corpus and Conv2D/self-attention specifics, the ARIA constraint as a monotonic cumulative-ratio bound (rather than just “dynamic alignment”), the three-stage Thinker post-training including on-policy distillation from text-conditioned responses, and Table 2’s per-concurrency latency numbers. For Luma’s voice-agent reading the streaming-numbers row alone is worth the file. It extends the Unified Multimodal Models cluster on the understanding-and-speech-output side (contrast with the joint A+V generation DiTs catalogued under Joint audio-video generation / SkyReels-V4: Multi-modal Video-Audio Generation, Inpainting and Editing model — Qwen3.5-Omni reasons over A+V input and emits text+speech, rather than generating synchronized A+V output), and complements the NEO-unify encoder-free direction (NEO-unify: Building Native Multimodal Unified Models End to End) as a different “native multimodal” stance: NEO-unify drops the VAE entirely; Qwen3.5-Omni keeps modality-specific encoders but trains AuT from scratch instead of bolting on Whisper. The ARIA single-stream-with-monotonic-ratio formulation is a small but reusable primitive — anyone training a streaming text+speech decoder where the two tokenizers have different rates can adopt the constraint without changing the rest of the stack.