Qwen3.5-Omni: A Native Omni-Modal Model with Thinker-Talker MoE Architecture

TL;DR

Qwen3.5-Omni is the omni-modal sibling of the Qwen3.5 family (Qwen3.5: Towards Native Multimodal Agents), extending the line from text+vision to text+image+audio+video input and text+speech output in a single model. It keeps the Qwen2.5-Omni Thinker-Talker split (a multimodal reasoning decoder feeds a streaming speech decoder) but upgrades both halves to Hybrid-Attention Mixture-of-Experts, replaces the Whisper-based audio encoder with a from-scratch Audio Transformer (AuT) pretrained on >100M hours of audio-video data, and ships in three sizes (Plus / Flash / Light) all with a 256K-token context window that maps to >10 hours of audio or ~400 s of 720p video at 1 FPS. Real-time interaction features (semantic interruption / turn-taking intent recognition, native function calling, native web search) are baked into the model rather than a separate orchestration layer. Plus is hosted-only on DashScope; Light is open-weights on HF.

Key claims

Native end-to-end omni-modal architecture — text, images, audio, and video are processed within a single computational pipeline, contrasted explicitly against “wrapper” pipelines that stitch separate encoders (e.g. Whisper for audio) onto a text backbone [§Architecture].
Thinker-Talker split: the Thinker is an autoregressive transformer decoder that handles multimodal reasoning and text generation; the Talker autoregressively predicts multi-codebook speech tokens that a Code2Wav renderer synthesizes frame-by-frame into streaming audio [§Architecture].
Both Thinker and Talker now use Hybrid-Attention Mixture-of-Experts, matching the broader Qwen3.5 family’s shift to sparse routing [§Architecture].
Audio encoder is a from-scratch Audio Transformer (AuT) — 128-channel mel spectrogram input, convolutional downsampling frontend, transformer trunk — replacing the Whisper-based encoder used in Qwen-Omni generations through Qwen2.5-Omni [§Architecture].
Pretrained on >100M hours of native multimodal audio-video data on top of text and visual corpora [§Pretraining].
256K-token context window across all three variants, equivalent to >10 hours of continuous audio input or ~400 s of 720p video at 1 FPS with audio [§Context].
Three Instruct variants — Plus, Flash, Light — covering hosted-only flagship through open-weights edge sizes [§Variants].
Real-time interaction stack baked into the model: TMRoPE (time-aware positional encoding aligning modalities), ARIA (Adaptive Rate Interleave Alignment dynamically aligning text and speech tokens), semantic interruption / turn-taking intent recognition distinguishing backchannels from genuine interruptions, native WebSearch and FunctionCall tool use, voice cloning [§Realtime API].
Speech recognition coverage 113 languages and dialects (up from 19 in the predecessor); speech generation 36 languages (up from 10) [§Multilingual].
Reported 215 SOTA results across audio, audio-visual understanding, reasoning, and interaction benchmarks for the Plus variant [§Benchmarks].
Audio-visual understanding reported on par with Gemini 3.1 Pro; text and visual benchmarks reported to match standard Qwen3.5 models — Qwen3.5-Omni-Plus stays close to non-thinking Qwen3.5-Plus on MMLU-Redux (94.2 vs 94.3), C-Eval (92.0 vs 92.3), IFEval (89.7 vs 89.7) [§Benchmarks].
Custom voice stability (Seed-zh, lower is better): 1.07 vs ElevenLabs 13.08, Gemini-2.5 Pro 2.42, GPT-Audio 1.11, Minimax 1.19 [§Voice Quality].
Distribution: Plus and Flash variants hosted on Alibaba Cloud’s DashScope API only; Light variant open-weights on Hugging Face [§Availability].

Method

The model splits perception and generation into two coupled stages. A single multimodal encoder accepts all input modalities — AuT for audio (mel spectrogram → conv frontend → transformer), a ViT-style encoder for images and video frames (similar to Qwen3-VL), and standard token embeddings for text — projected into a shared embedding space and concatenated into one input sequence. TMRoPE applies time-aware positional encoding so that audio and video tokens align on a common physical-time axis.

The Thinker is an autoregressive decoder over this concatenated multimodal sequence, generating text tokens. Both Thinker and Talker use Hybrid-Attention MoE: each token routes to a sparse subset of experts, with the attention mechanism able to weigh modality-specific token streams (e.g. focus on visual tokens during a video reasoning task) inside the same backbone. The Talker is a separate autoregressive component that consumes Thinker representations and emits streaming multi-codebook speech tokens, which Code2Wav renders into audio frame-by-frame. ARIA dynamically aligns the rates of text and speech tokens so the Talker can stream speech in parallel with text generation rather than waiting for the full text response.

For real-time conversation, the model is trained to emit a turn-taking-intent signal that classifies user audio inputs as backchannel (ambient noise, “uh-huh”) vs semantic interruption (the user wants the floor). Native WebSearch and FunctionCall follow OpenAI tool-format conventions via DashScope.

Results

Numbers below are Qwen’s own self-reports, aggregated from the launch blog and third-party recaps; no independent reproductions yet.

Audio (custom voice stability, Seed-zh, lower is better). Qwen3.5-Omni-Plus 1.07 vs ElevenLabs 13.08, Gemini-2.5 Pro 2.42, GPT-Audio 1.11, Minimax 1.19.
Text (Plus vs non-thinking Qwen3.5-Plus). MMLU-Redux 94.2 (vs 94.3), C-Eval 92.0 (vs 92.3), IFEval 89.7 (vs 89.7), LongBench v2 59.6, HMMT Nov 25 84.4 — multimodal expansion is reported not to cost text quality.
Audio-visual understanding. Reported “on par with Gemini 3.1 Pro” with 215 SOTA claims across audio + audio-visual + reasoning + interaction benchmarks; specific subscore tables not enumerated in the launch blog text accessible to the wiki.
Language coverage. ASR: 113 languages and dialects (up from 19). TTS: 36 languages (up from 10).
Context window. 256K tokens = ~10+ hours audio = ~400 s of 720p video at 1 FPS with audio, across all three variants.
Release timeline. March 30, 2026 (~6 weeks after the text-only Qwen3.5 main family that began Feb 16, 2026).
Distribution. Plus and Flash: DashScope API only. Light: open weights on HF (vLLM support expected to follow once weights land).

Why it’s interesting

This is the omni-modal completion of the Qwen3.5 release wave that Qwen3.5: Towards Native Multimodal Agents kicked off — the same family-wide Hybrid-Attention MoE backbone now wraps an audio encoder and speech decoder, closing the input/output set that the text+vision Qwen3.5 line left open. Two things make it worth tracking on Luma’s reading list. First, it’s the strongest open-or-API-accessible bid yet for the Thinker-Talker recipe (reasoning model emits text → streaming speech decoder synthesizes audio in parallel) as the right architectural template for voice-first agents — a different design point from the joint audio+video generation cluster (Joint audio-video generation / LTX-2: Efficient Joint Audio-Visual Foundation Model / MOVA: Towards Scalable and Synchronized Video-Audio Generation / SkyReels-V4: Multi-modal Video-Audio Generation, Inpainting and Editing model), which generates synchronized A+V output via dual-stream DiTs rather than reasoning over A+V input and emitting speech. Second, the AuT swap (Whisper out, custom 100M-hour-pretrained Audio Transformer in) is a concrete datapoint for “native multimodal” replacing the encoder-stitching pattern at scale — a different flavor of the same move that NEO-unify makes on the vision side (NEO-unify: Building Native Multimodal Unified Models End to End) and that the Qwen3.5 main family makes by retiring the separate Qwen3-VL line (Qwen3.5: Towards Native Multimodal Agents). All-Apache-2.0 status only applies to the Light variant; Plus is DashScope-only, which leaves the open-frontier voice-LLM story still narrower than the open text-VL story.