LongCat-Flash-Omni Technical Report

TL;DR

LongCat-Flash-Omni is Meituan’s open-source 560B-total / 27B-activated omni-modal model that takes any combination of text, image, video, and audio as input and emits text + streaming speech. Unlike the same team’s earlier LongCat-Next: Lexicalizing Modalities as Discrete Tokens, which tokenizes all modalities into a shared discrete vocabulary, this model keeps lightweight continuous encoders (~600M each) for vision and audio and bolts them onto the LongCat-Flash ScMoE backbone — trading a more standard early-fusion recipe for retention of the underlying LLM’s text capability. The technical contributions are the chunk-wise audio-visual interleaving scheme for real-time interaction (millisecond-level latency despite the 560B size), a 128K context for >8min real-time AV, and a Modality-Decoupled Parallelism training strategy that sustains >90% of text-only throughput on the omni-modal mix.

Key claims

560B total / 27B activated MoE built on LongCat-Flash, retaining ScMoE + zero-computation experts; vision encoder (LongCat-ViT, ~637M), audio encoder, and audio decoder (LongCat-Audio-Codec) are each ~600M [§2, Table 1].
LongCat-ViT supports native-resolution input (patch budget 576–5832 during training) and uses 2D-RoPE + SwiGLU + RMSNorm + QK-norm; trained from scratch on 14.6B contrastive image-text samples with progressive resolution + progressive modality (video added last) curriculum [§2.1].
Audio is handled two ways across training: pretraining stages 1–4 use a 4-codebook 16.67 Hz discrete tokenizer (LongCat-Audio-Codec, one semantic + three acoustic codebooks), and from stage 5 onward a continuous Fbank-based streaming encoder replaces it for input — the model keeps emitting the 4-codebook discrete tokens for output regardless [§2.2].
Streaming audio is achieved with an Fbank → Pre-FFN downsample (8×, 80ms frames) → FSMN-layer streaming encoder; only the final six layers carry a 1-frame look-ahead, the rest are strictly causal [§2.2].
Audio is reconstructed directly by the LongCat-Audio-Codec decoder (LSTM + conv + causal transposed conv, GAN-trained) — no diffusion/flow-matching vocoder — with only 3-frame look-ahead, chosen explicitly for low-latency real-time use [§2.2].
Real-time AV interaction uses a chunk-wise audio-visual feature interleaving template: <timestamp>:<video-tokens><audio-start><audio-tokens><timestamp>:<video-tokens><audio-tokens>…<audio-end>; textual timestamps are prepended to each video frame to give the model temporal awareness [§2.4.1, §2.4.2].
Sparse-Dense Sampling: 2 FPS / 1-second chunks during the user-input period, 0.5 FPS / 2-second chunks buffered during the model’s response and prepended to the next user turn — a deliberate asymmetric video sampling rate during turn-taking [§2.4.2].
128K context window supports >8min real-time AV interaction with long-term memory and multi-turn temporal reasoning [§1].
Modality-Decoupled Parallelism (MDP) optimizes the LLM, vision encoder, and audio encoder independently for performance and memory; reported to sustain >90% of pure-text training throughput on the omni mix [§1, §5].
Pretraining corpus is >2.5T tokens spanning audio (speech-text interleaved + understanding), generic image-text (cleaned + re-captioned with a SigLIP threshold + concept-resampled with a 200K Chinese lexicon, MetaCLIP-style), interleaved image-text, OCR / grounding / GUI, STEM, multi-image, video, and long-context multimodal data [§3.1].
Claims state-of-the-art on omni-modal benchmarks (Omni-Bench, WorldSense) among open-source models, with competitive results on per-modality text / image / video understanding and speech understanding + generation [§1, §7].
Training strategy is “early-fusion with curriculum”: text-pretrained base → progressively add audio and visual data → balanced multimodal mixture, with the explicit design criterion that no unimodality regresses [§1, §3.2].

Method

The architecture is a four-piece composition: (1) LongCat-ViT vision encoder with native-resolution patchification, 3D convolution (temporal stride 2) for video, and an interpolative downsample on visual tokens if a per-frame budget is exceeded; (2) a streaming Fbank+FSMN audio encoder with one-frame look-ahead only in its last six layers; (3) the LongCat-Flash MoE LLM backbone (560B total, 27B active, MLA attention, ScMoE, zero-computation experts) which consumes the interleaved chunked features and emits text + 4-codebook audio tokens; (4) the LongCat-Audio-Codec decoder which converts emitted audio tokens to waveform with only 3-frame look-ahead. A two-layer pre-normalized MLP with 2× pixel-unshuffle aligns vision features to the LLM hidden space.

For real-time AV interaction, audio and video are time-aligned and chunked, then interleaved by timestamp before being fed to the LLM. During input, a denser sampling (1s chunks, 2 FPS) preserves information; during model response, a sparser buffer (2s chunks, 0.5 FPS) is prepended to the next user turn — explicitly to keep visual context across turn-taking without paying full latency cost. The 128K context window is the budget within which the >8-minute real-time interaction lives.

Training is staged. Pretraining stages 1–4 use the discrete 4-codebook audio tokenizer at the input side, enabling consistent next-token-prediction; from stage 5, the discrete audio input is replaced with the continuous Fbank+FSMN encoder to recover fine-grained acoustic detail that discretization removes. Output remains 4-codebook discrete tokens throughout, aligned with next-token prediction. Modality-Decoupled Parallelism configures the LLM, vision encoder, and audio encoder independently to handle their different compute/memory profiles.

Results

State-of-the-art on Omni-Bench and WorldSense among open-source models at filing time [§1, §7].
Audio-visual response latency is millisecond-level end-to-end despite the 560B total parameter count — enabled by the 600M-class lightweight encoders/decoder, 3-frame audio look-ahead, FSMN-based streaming, and chunk-wise interleaving [§1, §2.2, §2.4].
Modality-Decoupled Parallelism sustains >90% of text-only training throughput on the omni mix [§1, §5].
The model claims competitive performance on per-modality benchmarks (text, image understanding, video understanding, speech understanding, speech generation) while also winning omni-modal benchmarks — explicitly tested against the “unimodal regression” failure mode [§1, §7].
Specific per-benchmark numbers (Omni-Bench score, WorldSense score, ASR WER, speech-naturalness MOS) are reported in §7 but were truncated at fetch time at filing.

Why it’s interesting

Read alongside LongCat-Next: Lexicalizing Modalities as Discrete Tokens (the same Meituan team’s discrete-everywhere “DiNA” omni at 68.5B/3B-active), LongCat-Flash-Omni is the opposite design bet from the same vendor at 8× larger active parameter count: continuous vision encoder, continuous audio encoder, discrete audio output only, real-time AV as a first-class capability — versus DiNA’s shared discrete vocabulary for all modalities and a modality-agnostic backbone. The two together bracket Meituan’s hedge across the continuous-vs-discrete UMM axis tracked on Unified Multimodal Models, and the choice to keep encoders continuous in the larger, real-time-focused model is itself evidence about which side scales better when latency matters.

The chunk-wise AV interleaving + sparse-dense turn-taking sampling extends the design vocabulary on Joint audio-video generation — most filed JAV papers (MOVA, SkyReels-V4, OmniForcing, MultiTalk) handle joint generation rather than the bidirectional streaming-input + streaming-output regime LongCat-Flash-Omni targets. The model is also the largest filed open omni release on Open foundation-model releases by both total (560B) and active (27B) parameter count, and it inherits the ScMoE + zero-computation-experts routing recipe that the MoE Routing Design page tracks; the modality-decoupled parallelism scheme is the systems-side counterpart to MegaScale-MoE’s drop-TP recipe, applied to a heterogeneous-modality training mix rather than pure text.