LongCat-Next: Lexicalizing Modalities as Discrete Tokens

TL;DR

LongCat-Next is Meituan’s open-source attempt at a fully discrete native multimodal foundation model: text, vision, and audio are all tokenized into a shared discrete codebook and trained under a single next-token-prediction objective on the LongCat-Flash-Lite MoE backbone (68.5B total / 3B active). The headline contribution is DiNA — Discrete Native Autoregression — instantiated through a vision tokenizer (dNaViT) and audio tokenizer that the team designed to reach semantic completeness (lossless enough for both understanding and generation) using Semantic-and-Aligned Encoders + Residual Vector Quantization. The team claims this breaks the long-standing performance ceiling of fully-discrete vision modeling: LongCat-Next reportedly surpasses Qwen3-Omni on unified-model benchmarks, beats specialized Qwen3VL-A3B on visual understanding, and competes with Flux-dev on image generation (especially text rendering), while audio matches Gemini 3.1 Flash-Lite preview and MiMo-Audio. Weights, tokenizers, and tech report are open on GitHub / Hugging Face.

Key claims

DiNA represents text, vision, and audio in a single shared discrete token space and trains them under one next-token-prediction objective with a modality-agnostic MoE backbone (LongCat-Flash-Lite, 68.5B total / 3B active) plus modality-specific tokenizer/detokenizer pairs; no per-modality heads in the backbone [§1, §2.1, Fig. 3].
The framework formalizes semantic completeness as a discretization criterion: for any image-centric query Q, the discrete representation z should satisfy P(A|z,Q) ≈ P(A|I,Q), decomposed into discriminative invariance (preserves semantic attributes for understanding) and generative sufficiency (preserves the content needed for faithful reconstruction) [§2.2.1, Eq. 1].
A class of Semantic-and-Aligned Encoders (SAE) — language-aligned encoders like QwenViT / MoonViT / AIMv2 — is identified as a strong pre-quantization space because large-scale VL alignment training produces representations that are simultaneously semantically rich and “language-affine”; the SAE’s residual architecture is observed to also preserve a latent low-level signal pathway, even without explicit reconstruction supervision [§2.2.1, §2.2.2].
dNaViT (Discrete Native Any-resolution Vision Transformer) uses Residual Vector Quantization on top of an SAE to build hierarchical discrete tokens that preserve both high-level semantics and fine-grained details, supporting any-resolution tokenization and de-tokenization at up to 28× compression [§2.2, §2.2.5, Fig. 3].
Multi-level RVQ tokens are exposed to the autoregressive backbone via additive encoding + a DepthTransformer for efficient decoding, unlocking an exponential representation space across the residual hierarchy while keeping the AR step a single backbone forward pass [§1, §2.2.5].
Audio is tokenized at 12.5 Hz via an RVQ-based pipeline over a Whisper encoder; reconstruction uses a paired decoder + flow-matching refinement network for high-fidelity de-tokenization [§1, §2.3].
A unified audio training paradigm aligns segment-level text and audio tokens with stochastic delays, enabling both parallel and serial text-guided speech generation in one model [§1, §2.5].
LongCat-Next is claimed to surpass Qwen3-Omni on unified-model benchmarks, outperform the specialized Qwen3VL-A3B on visual understanding, and compete with Flux-dev on image generation — particularly in text rendering [§1, Fig. 1].
Audio benchmarks: LongCat-Next is reported to outperform both omni and speech-specialized models of comparable parameter scale, with Gemini 3.1 Flash-Lite preview and MiMo-Audio called out as the relevant baselines [§1, §3.1.3].
The unified discrete framework is claimed to reconcile the long-standing understanding-vs-generation conflict in fully-discrete VLMs: under DiNA, the two are reformulated as the same predictive process with different conditional priors (image tokens condition text generation; text tokens condition image generation) [§1, §3.2.3].
Multimodal training does not degrade foundational language capabilities — adding vision and audio is reported as complementary rather than a trade-off (the paper lists “modality synergy instead of compromise” as an explicit design criterion and claims it holds empirically) [§2, §3.1.4].
The system is trained on >2T tokens at A3B (68.5B total) scale using a VHalf-based pipeline-parallelism infrastructure [§1, §5.1].

Method

The architecture (Fig. 2) is structurally simple: modality-specific tokenizer / de-tokenizer pairs convert raw inputs to and from discrete IDs; a modality-agnostic LongCat-Flash MoE backbone consumes the resulting unified token sequence under a single next-token-prediction objective. There are no separate vision or audio heads in the backbone.

Vision pipeline (dNaViT). The pre-quantization space is a Semantic-and-Aligned Encoder (SAE) — a vision encoder pretrained with large-scale language-aligned supervision over diverse image-centric tasks (captioning, OCR, QA, visual reasoning). The paper observes that the residual structure of such encoders implicitly preserves low-level signal alongside the semantic features, even without reconstruction supervision. dNaViT quantizes the SAE output with Residual Vector Quantization (RVQ) — hierarchical residual codebooks where each level encodes “the residual of the residual” — producing multi-level discrete IDs that the AR model consumes via additive encoding + a lightweight DepthTransformer for decoding. The same tokenizer supports both directions (image → IDs for understanding, IDs → image for generation), at arbitrary resolution and up to 28× compression.

Audio pipeline. A Whisper-encoder + RVQ stack tokenizes speech/audio at 12.5 Hz; a paired decoder with a flow-matching refinement net handles de-tokenization. For training, segment-level text and audio tokens are aligned with stochastic delays so the model learns both parallel and serial text-guided speech generation.

Backbone & training. LongCat-Flash-Lite is a Mixture-of-Experts decoder-only Transformer (A3B / 68.5B total / 3B active per token), inherited from Meituan’s prior LongCat-Flash work, and is left modality-agnostic so it can be a multi-task learner across language, visual understanding, visual generation, audio comprehension, and audio synthesis under one autoregressive objective. Visual training is staged: pre-training, then a cluster-based rebalancing mid-training stage, then supervised fine-tuning. Total training corpus is >2T tokens. Infrastructure uses a VHalf-based pipeline parallelism scheme.

Results

Visual understanding. Claimed to outperform the specialized Qwen3VL-A3B on visual understanding benchmarks and to surpass existing unified frameworks like Qwen3-Omni [§1, §3.1.1]. The paper also runs an “Information Recovery Analysis” arguing the discrete representation now matches continuous representations on understanding tasks — directly addressing the “performance ceiling of discrete vision” concern [§3.2.1, §3.2.2].
Visual generation. Competitive with Flux-dev under a 28× compression ratio, with text rendering explicitly highlighted as a strength [§1, §3.1.2]. Generation is supported at arbitrary resolution.
Audio. Outperforms omni baselines (Gemini 3.1 Flash-Lite preview) and speech-specialized baselines (MiMo-Audio) at comparable parameter scale; supports low-latency audio-to-audio interaction and voice cloning [§1, §3.1.3].
Text. Foundational language capabilities are reported to be preserved (i.e. no measured regression from adding vision and audio) — the paper frames this as evidence that modality synergy beats modality compromise in this regime [§3.1.4].
Analysis. §3.2 includes a Platonic-representation-hypothesis-style measurement of cross-modal representation alignment in the modality-agnostic MoE (§3.2.6), a study of training dynamics for modality-agnostic MoE routing (§3.2.5), and a semantic comparison of parallel vs serial audio generation (§3.2.4).
Specific benchmark tables (per-task scores against Qwen3-Omni, Qwen3VL-A3B, Flux-dev, Gemini 3.1 Flash-Lite, MiMo-Audio) are referenced in the report’s §3 but are not transcribed here at filing time.

Why it’s interesting

LongCat-Next is the clearest fully-discrete native unified-multimodal-model release filed on the wiki so far, and complements the other UMM paradigms tracked in Unified Multimodal Models from a sharp angle: where NEO-unify: Building Native Multimodal Unified Models End to End (NEO-unify) goes encoder-free with continuous pixel I/O and an MoT split between understanding and generation, LongCat-Next goes the opposite direction — everything is a discrete token, and the backbone is modality-agnostic — and reports it now competes with continuous-representation specialists on understanding. The semantic-completeness framing + SAE-based RVQ recipe is a concrete proposal for the wiki’s open question on what capacity floor a UMM needs to host both objectives; it should be tested against the UniG2U-Bench finding (UniG2U-Bench: Do Unified Models Advance Multimodal Understanding?) that unified training usually hurts understanding versus the base VLM.

The other contrast worth tracking: Ming-flash-omni 2.0 (Ming-flash-omni 2.0) is a recent omni-MLLM at roughly the same scale that goes the continuous-AR + DiT-head route for audio and a separate native multi-task head for image manipulation — LongCat-Next is the discrete-everywhere counter-recipe at the same capability surface (see / paint / speak in one model). Together they bracket the design space for “unified open omni at A3B–10B-active scale” with two opposite bets on whether vision/audio should stay continuous or be tokenized. For Open foundation-model releases, this is also another data point in the single-model + companion-tokenizers + tech-report + HF/GitHub packaging pattern that the wiki has been tracking, with a coordinated PDF tech report, GitHub repo, HF weights, and a live demo at longcat.chat.