Kwai Keye-VL Technical Report

TL;DR

Kwai Keye-VL is an 8B multimodal LLM from Kuaishou’s Keye team built specifically for short-form video understanding (the TikTok / Kuaishou content surface), with Qwen3-8B as the language decoder and SigLIP-400M-384-14 as the ViT, connected by an MLP projector and unified through 3D RoPE so position encodings map to absolute time. The training story is the contribution: a 600B-token mostly-video pretraining corpus filtered with CLIP scores + VLM discriminators + MinHash dedup, run through a four-stage pretraining curriculum (vision pre-training → cross-modal alignment → multi-task pretraining → annealing with model merging) followed by a two-phase post-training pipeline (SFT/instruction-following → reasoning RL). The team also releases KC-MMBench, a new benchmark for real-world short-video scenarios where Keye-VL claims a significant lead at its scale, and reports it matches general-purpose VLM benchmarks while leading on video-centric ones [§Abstract, Fig. 1].

Key claims

The architecture is a classic three-stage MLLM (ViT → MLP projector → LLM) with 3D RoPE applied uniformly across text, image, and video tokens, where the temporal axis of the 3D RoPE establishes a one-to-one correspondence between position encoding and absolute time so that frame rates / sampling intervals are recoverable from the embedding alone [§Architecture].
Native dynamic resolution is preserved by tiling each image into 14×14 patches at its original aspect ratio and using 1D interpolation + 2D RoPE in the ViT; ablation reports this beats 1D-interpolation-only on high-resolution images [Table 2].
The training data is video-heavy at >600B tokens, with three quality controls applied to the entire corpus: CLIP-score filtering, VLM-based discriminators for content quality, and MinHash deduplication against eval sets to prevent leakage [§Data].
Internal “self-built” datasets are constructed specifically to fill open-source gaps in detailed image captions and Chinese OCR — the paper treats these as a necessary complement to public data rather than a substitute [§Data].
Pre-training is a four-stage progressive curriculum: Stage 0 continues pretraining the ViT on the internal distribution to handle dynamic resolution; Stage 1 freezes the backbone and trains only the MLP projector for cheap image-text alignment; later stages do multi-task pretraining (dense captioning, OCR, VQA) and an annealing phase that merges checkpoint weights [§Training].
Post-training is split into a non-reasoning SFT phase (instruction following, alignment) and a reasoning RL phase that “stimulates advanced reasoning” — the paper frames these as orthogonal capability axes that benefit from being optimized separately [§Post-training].
KC-MMBench is released alongside the model as a new benchmark tailored to real-world short-video scenarios, where Keye-VL is reported to show a “significant advantage” over similar-scale models [§Abstract, §Benchmark].
Comprehensive human evaluations are reported to confirm the user-facing experience is superior to other ~8B-scale leading models — i.e. the paper claims an automated-benchmark + human-eval double win at its scale [§Abstract, Fig. 1].
Model and code are released openly: weights on Hugging Face (Kwai-Keye/Keye-VL-8B-Preview), code on GitHub (Kwai-Keye/Keye), with day-0 swift and vLLM serving support added shortly after release [§Released artifacts, model card].

Method

The architecture is conventional for a current-generation MLLM — SigLIP-400M-384-14 ViT, MLP projector, Qwen3-8B decoder — and the novelty is in three places. (1) Video handling uses 3D RoPE where the temporal axis is tied to absolute timestamps rather than frame indices, so the model can in principle reason about frame rate, sampling stride, and event spacing without conditioning on it explicitly. (2) The vision tower is run at native dynamic resolution: each input image is patched into 14×14 tiles at its original aspect ratio, and a 1D + 2D RoPE combination handles the variable-shape position encoding. (3) The pre-training pipeline is structured around progressive unfreezing — Stage 0 only adapts the ViT to the internal data distribution, Stage 1 only trains the MLP projector (cheap image-text alignment), and later stages train end-to-end with the full multi-task objective followed by a model-merging annealing phase. Post-training is then a two-phase pipeline: a non-reasoning SFT phase for instruction following (similar to standard MLLM SFT), and a separate reasoning RL phase that targets advanced reasoning capabilities. The paper frames KC-MMBench, the released eval benchmark, as the missing piece — existing video-MLLM benchmarks (Video-MME, MVBench, etc.) don’t capture the dynamic, information-dense character of real Kuaishou/TikTok-style short videos, and KC-MMBench is constructed to fill that gap.

Results

The headline result is a Pareto-style claim from Figure 1: Keye-VL-8B sets a new SOTA among similar-scale models on video-centric benchmarks (KC-MMBench plus the standard suite) while remaining competitive on general perception and reasoning benchmarks where prior short-video specialists usually regress. Quantitative deltas are not reproduced verbatim here because the paper body was not directly accessible at filing — see the technical report PDF for per-benchmark tables. Ablations confirm two pieces of the design: 2D-RoPE on the ViT outperforms 1D interpolation alone on high-resolution images [Table 2], and the staged SFT→RL post-training is what enables the reasoning-benchmark gains over an SFT-only baseline (the same pattern reported by Video-Thinker — SFT alone is a tax, RL is what generalizes).

Why it’s interesting

To answer Jiaming’s prompt directly (“Wonder if it does anything interesting on videos”): the most distinctive video-specific choice here is tying 3D RoPE to absolute timestamps rather than frame indices, which is the position-encoding side of the Reasoning RL + video-grounding recipe that Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning approaches from the data side. Keye-VL is also a clean comparison point for the Open foundation-model releases pattern at the 8B-MLLM tier: it’s a Kuaishou-internal model that ships with weights, a tech report, code, day-0 vLLM/swift support, and an accompanying released benchmark (KC-MMBench), and is the natural counterpart to Qwen3-VL on the video-understanding axis — same backbone family (Qwen3-8B), same ViT lineage (SigLIP), but optimized for the short-video content surface rather than long documents. Three things make it worth tracking: (a) the 600B-token mostly-video pretraining corpus is far larger than what most open video MLLMs are trained on, and the filtering/dedup recipe is a Synthetic Training Data-adjacent contribution; (b) the four-stage progressive-unfreeze pretraining + model-merging annealing is unusually explicit about staging; (c) the v1.5 follow-up (Sep 2025, arXiv:2509.01563, not separately filed here) introduces a Slow-Fast video encoder that decouples spatial resolution from temporal coverage and extends context to 128K — useful to track as a follow-up since several Luma-relevant questions (long-context video, spatial-temporal tradeoff in encoders) sit on that axis.