Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Ranking

TL;DR

Joint technical report for the Qwen3-VL-Embedding and Qwen3-VL-Reranker model series — already filed at Luma via the HuggingFace model-card forms (Qwen3-VL-Embedding-8B and Qwen3-VL-Reranker-8B). This page covers what the technical report adds beyond those cards: the multi-stage training pipeline (s0→s3, contrastive pre-training through reranker-distillation refinement), the exact video tokenization budget (~1 FPS, 4,500-token cap across all frames), the MRL + QAT training objective details, and the ablations on visual granularity, stage progression, MRL, and embedding quantization. The report’s headline number is unchanged: Qwen3-VL-Embedding-8B reaches 77.8 on MMEB-V2, ranked first as of January 8, 2026 [Abstract].

Key claims

The embedding model is dual-tower with [EOS]-token last-layer pooling; the reranker is single-tower cross-encoder with cross-attention — i.e. the standard recall→rerank decomposition, both stages on the same Qwen3-VL backbone [§Architecture].
Training is multi-stage (s0→s3), progressing from large-scale contrastive pre-training to reranker-model distillation, ablated per-stage in the report [§Training, RQ7].
The embedding training objective extends the Qwen3-Embedding loss with Matryoshka Representation Learning (variable output dimensions) and Quantization-Aware Training (multiple numerical precisions) [§Training Objective].
Video tokenization uses fps=1.0 as the default sampling rate, and the total token budget across all frames is constrained to 4,500 tokens (~9.2M pixels) — this is the video budget across the entire clip, not per-frame [§Video Tokenization; GitHub default].
Input context window is 32k tokens; multilingual coverage of 30+ languages is inherited from Qwen3-VL [Abstract].
Qwen3-VL-Embedding-8B attains 77.8 overall on MMEB-V2, ranked first among all models as of January 8, 2026; results are consistent across image, visual-document, and video retrieval subtasks [Abstract, §Results].
The report explicitly studies (RQ5) the MRL + embedding-quantization trade-off on retrieval performance, index storage, and retrieval latency, and (RQ6) the effect of visual granularity (spatial resolution + temporal granularity) across domains [§Ablations].

Method

The embedder takes a single- or mixed-modal input (text, image, screenshot, document image, video, or any combination), passes it through the Qwen3-VL backbone, and reads out the last-layer hidden state at the [EOS] token as the embedding. Vision and video share Qwen3-VL’s standard patch + merger pipeline (no separate CLIP-style tower).

Two representation-shaping techniques sit on top of an extended contrastive loss: MRL trains the embedding so that prefixes of the vector remain valid embeddings, enabling deployment-time truncation (e.g. 4096→512 with documented small NDCG loss). QAT trains the model to remain accurate under post-hoc quantization of the output vector (int8 / int4 / binary deployment). The reranker is a single-tower cross-encoder scoring a (query, document) pair directly, optimized with the Qwen3-Reranker objective.

The training pipeline runs in stages s0–s3: s0 is large-scale weakly-supervised contrastive pre-training over multimodal pairs; s1 adds supervised fine-tuning on labeled retrieval data; s2/s3 refine the embedder against the trained reranker via distillation. The report ablates each stage’s contribution.

Video specifics. The default sampling rate is 1 FPS, and the total token budget across all sampled frames is capped at 4,500 tokens (~9.2M pixels per clip, shared globally). The GitHub reference code documents fps=1.0 as the default and notes the 4,500-token / ~9.2M-pixel global cap. For a 16-frame clip at the documented 1280×768 max per frame, the 4,500-token cap is the binding constraint across all frames — i.e. the per-frame budget shrinks as the clip length grows.

Results

The report’s headline numbers match the model-card blogs already filed: 77.8/77.9 overall on MMEB-V2 for the 8B variant [Abstract; also reported as 77.9 in the HF card], ranked first across the benchmark’s text/image/video/visual-doc subtasks at release. Per- subtask figures (80.1 image, 66.1 video, 83.3 visual-doc for 8B) are detailed on the model-card page Qwen3-VL-Embedding-8B and not re-tabulated here.

What the report adds beyond the cards:

Stage progression (RQ7): per-stage MMEB-V2 numbers tracking the embedder from s0 (contrastive only) through s3 (post-reranker distillation); the largest jumps come from the reranker- distillation stages.
Visual-granularity ablation (RQ6): spatial resolution and temporal granularity (FPS, frame count) ablated independently across image / document / video subtasks; the video subtasks are predictably the most sensitive to temporal granularity.
MRL + quantization ablation (RQ5): NDCG-vs-storage and NDCG-vs-latency curves across MRL dimensions and quantization formats; a companion fine-tune (tomaarsen/Qwen3-VL-Embedding-2B-vdr) shows the released model stays within 0.3% of peak from 2048 dims down to 512 and holds 92% at 64 dims (32× compression).

Why it’s interesting

Jiaming’s framing — “dataset search / taxonomy analysis” — is the natural Luma application. A single embedder that places text queries, keyframes, and (low-FPS) video clips in one space is the substrate for unsupervised corpus clustering and content search. Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training showed that text-side data-mixture curation reduces to embed→cluster→ quality-filter→merge; a competent multimodal embedder makes the same loop possible on video corpora without first running a captioning pass.

Three caveats are real and visible in the report.

Video temporal resolution is coarse. Default fps=1.0 with a 4,500-token global budget across all frames means anything past coarse content matching — action segmentation, motion dynamics, fine-grained temporal structure — is unlikely to be preserved. This is fine for “find clips that look like this clip” but a poor fit for “find clips where X happens at second 12.”
Smallest model is 2B. Even with INT8 + MRL truncation, the model is significantly heavier than a domain-specific tagger (FlashTagger) and is probably wrong for high-throughput online labeling. The likely deployment shape is offline / batch dataset analysis, not online tagging.
The training mix is caption-heavy. Performance is reported on benchmarks with paired text + image / video, where the text side carries a lot of the signal. On uncaptioned raw web video the embedding quality will degrade — exactly the regime where dataset-taxonomy work would want the model to perform.