Kling-Omni Technical Report

TL;DR

Kling-Omni (Kling-O1) is Kuaishou’s generalist video-generation framework that takes “multimodal vision-language” (MVL) — interleaved text, reference images, and video contexts — as input and produces cinematic-quality video for generation, editing, and reasoning tasks in a single end-to-end stack. The system has three components: a Prompt Enhancer (MLLM, SFT + RL-tuned for factual correctness and training-distribution alignment), an Omni-Generator DiT trained with pre-training → SFT (continue-training + quality-tuning) → DPO → two-stage distillation (PCM-style trajectory matching + TDM-style ODE distribution matching), and a Multimodal Super-Resolution module with shifted-window local attention and asymmetric KV-cacheable cross-attention. Training is run on a 3D-parallel system with elastic Ulysses parallelism, MM-FlashAttention with arbitrary cross-modal masks, and a 97% Effective Training Time Ratio. The paper introduces OmniVideo-1.0, a >500-case GSB-style human benchmark against Veo 3.1 and Runway Aleph, and reports that Kling-Omni is the only entry supporting element-library reference, multi-image editing, subject-driven editing, and previous-shot generation simultaneously.

Key claims

Multimodal Vision Language (MVL) is proposed as a unified input representation that combines natural language as a “semantic skeleton” with reference images and video contexts treated as a cohesive language, replacing static text encoders [§1, §2.1].
The architecture is decoupled into three trained modules — Prompt Enhancer (MLLM) → Omni-Generator (DiT) → Multimodal Super-Resolution (cascaded DiT) — rather than a single end-to-end model [§2.1].
Distillation compresses the Omni-Generator from 150 NFE to 10 NFE in two stages: (1) phase-wise trajectory-matching distillation (PCM / HyperSD-style), with the student executing the target 10-step scheduler directly rather than passing through an intermediate-NFE state, and (2) distribution-matching distillation via TDM-style ODE sampling, keeping the trajectory-matching objective as a regularizer [§2.2.4].
DPO is preferred over GRPO for RL alignment because it avoids GRPO’s expensive trajectory sampling, exploiting the one-step diffusion forward [§2.2.3].
The Prompt Enhancer is trained in two phases (SFT to install a reasoning chain, then RL with a reward over factual correctness, content richness, semantic plausibility, and similarity to training-data distribution) on a specialized multilingual dataset [§2.3].
Multimodal Super-Resolution uses shifted-window local attention in odd layers and asymmetric attention where condition tokens are restricted to self-attention while noisy tokens attend to the full sequence — enabling KV-cache reuse of condition features across sampling steps [§2.4].
Training infrastructure: microbatch-level elastic Ulysses-parallel switching with an online adaptive scheduler over DP ranks, two-tier intra/inter-node all-to-all to relieve spine switches, a packing MM-FlashAttention operator supporting arbitrary cross-modal masks and variable-length sequences in one kernel, pipeline-aware activation offloading, and a virtual-pipeline-stage-aware activation-reuse mechanism [§2.5.1, §2.5.2].
97% Effective Training Time Ratio reported, achieved via RDMA-traffic-based hang detection within a minute, sub-minute restarts, NVMe artifact concurrent loading, parallelized NCCL/kernel warmup overlapped with I/O, and pre-shuffled Parquet sequential reads [§2.5.3].
Inference is FP8-quantized (most GEMMs and self-attention), with quantization fused into other kernels for zero-overhead, FP8 communication, and a cache scheme over long reference-image/video conditional inputs giving roughly 2× speedup, with a cache-offload path for memory pressure [§2.6].
Data is sourced via real-world acquisition plus task-oriented synthetic construction; reverse-synthesis is used to build reference-to-video samples that preserve natural video temporal consistency while providing explicit control [§3.1].
OmniVideo-1.0 is introduced as a >500-case evaluation set covering subject categories (humans / cartoons / animals / clothing / props), application scenarios (professional production / e-commerce / social media), and challenges (complex actions, wide-angle perspectives, cross-style integration, multi-element fusion), scored on Dynamic Quality, Prompt Following, Identity Consistency, and Video Consistency [§4.1.1, §4.1.2].
Capability-coverage comparison (Table in §4.1): only Kling-Omni supports element-library reference, image+element-library reference, replacement editing, stylization, attribute manipulation, special effects, video matting, multi-image editing, subject-driven editing, and next-shot/prev-shot generation simultaneously; Veo 3.1 supports a strict subset (image ref + add/remove); Runway Aleph supports most editing primitives but no element-library reference or next-shot generation [§4.1, Capability table].

Method

Kling-Omni is a three-stage cascade. The Prompt Enhancer (PE) is a fine-tuned MLLM that maps user inputs onto the training-data distribution: SFT installs a chain-of-thought “thinking process” and RL rewards factual correctness, content richness, semantic plausibility, and proximity to the high-quality training prompt distribution. PE outputs become MVL conditioning for the Omni-Generator, a DiT that processes text and visual tokens in a shared embedding space. The Omni-Generator’s training is itself a four-stage pipeline: pre-training on text-video pairs with mixed-length captions (concise → narrative) and image-to-video tasks blended in; SFT in two sub-stages (Continue-training on heterogeneous interleaved image+video+text tasks including reference-to-video, image/video editing, and semantic-understanding tasks; Quality-tuning on a curated balanced-task premium dataset); DPO with diverse MVL conditions sampled into preference pairs scored by human evaluators on motion dynamics and visual integrity; and two-stage distillation (PCM-style trajectory matching with phase-wise scheduler partitioning where the student directly targets 10-NFE sampling, followed by TDM-style ODE distribution matching with the trajectory objective preserved as regularization). The Multimodal Super-Resolution cascade is initialized from the base DiT weights, conditioned on both low-resolution latents and the full MVL signal; it replaces full attention with shifted-window local attention and uses asymmetric attention so condition KV is cacheable across sampling steps.

The training system is built around three pieces. Data: a central scheduler assigns samples to DP groups for balanced workloads and dynamically partitions tokens across PP stages for VAE/text-encoder inference. Parallelism: microbatch-level elastic Ulysses switching, predetermined by an online adaptive scheduler with asynchronous pipeline, plus a two-tier intra-/inter-node all-to-all to keep spine switches lightly loaded. Kernels: a packing MM-FlashAttention operator supporting arbitrary cross-modal masks and variable-length sequences within a single kernel, plus selective activation recompute, pipeline-aware CPU offloading, and virtual-pipeline-stage-aware activation reuse across model chunks with identical inputs. Inference adds hybrid Ulysses + tensor parallelism with computation–communication overlap, FP8 quantization with fused (de)quant kernels and FP8 communication, and a cache + cache-offload scheme tailored to long reference-image/video conditional inputs.

Results

OmniVideo-1.0 GSB human evaluation (Fig. 7, headline panel) is the primary reported result against Google Veo 3.1 and Runway Aleph; specific GSB win-rates and absolute scores are in the truncated tail of the paper and not surfaced in this filing.
Capability matrix [§4.1, Table]: Kling-Omni is the only system among the three that ticks every capability slot — element-library reference, image+element ref, replacement, stylization, attribute manipulation, special effects, matting, multi-image editing, subject-driven editing, and prev-shot generation; Veo 3.1 supports only image reference and add/remove; Runway Aleph supports most editing primitives and prev-shot generation but no element-library reference.
Distillation: 150 NFE → 10 NFE compression of the Omni-Generator [§2.2.4]; no headline FID / VBench numbers are surfaced in the filing.
Inference cache: roughly 2× speedup attributed to caching long conditional inputs [§2.6].
Training reliability: 97% Effective Training Time Ratio over the run, recovery time compressed to minute-level [§2.5.3].
Quantitative ablation tables (DeepStack depth analogues, per-task scores, per-metric breakdowns of OmniVideo-1.0) appear in §4 beyond the fetched window and are not surfaced here — open for follow-up read.

Why it’s interesting

Kling-Omni is the closest filed analogue to OmniWeaving: Towards Unified Video Generation with Free-form Composition and Reasoning in scope and architecture: both build an MLLM-conditioned video DiT that unifies generation + reference-based composition + editing as one stack, both rely on a synthetic-data-heavy curriculum, and both wave at “thinking” as a way to bridge instruction interpretation and generation. Kling-Omni’s distinction is the system-engineering depth — explicit FP8 quantization, MM-FlashAttention packing, elastic Ulysses, two-tier all-to-all, 97% ETTR — which puts it next to FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling and the Scaling and Optimizing Frontier Model Training (Fireworks AI) framing of frontier-scale training as much as next to other unified video models. The distillation choice (PCM trajectory matching → TDM-style ODE distribution matching, both running through the trajectory regularizer) lands directly inside the Diffusion Distillation design space and is the closest filed instance of “skip the intermediate-NFE student stage and target the final NFE directly” — a contrast with Phased DMD: Few-step Distribution Matching Distillation via Score Matching within Subintervals which keeps explicit phased stages. As a closed-source flagship-vs-flagship comparison (Kling-Omni vs Veo 3.1 vs Runway Aleph), it is a useful market datapoint alongside Introducing Runway Gen-4.5 and Veo 3 Tech Report, even though no head-to-head numbers are released and OmniVideo-1.0 is a self-introduced benchmark with no independent calibration.