The Many Senses of Visual Similarity: A Text-Prompted Image Perceptual Metric

TL;DR

TPIPS is a text-prompted perceptual similarity metric — the same triplet-comparison protocol as LPIPS / DreamSim, but with each triplet annotated multiple times conditioned on a specific aspect (color, pose, lighting, background, camera angle, etc.), producing a metric whose score depends on which visual axis the user asks about. The authors collect ~1M human judgments over 25K image triplets across 260K triplet-aspect combinations from T2I-generated images, plus an out-of-distribution 2AFC set from four real vision/graphics pipelines (image editing, compositing, novel-view synthesis, image-to-3D). They benchmark 15+ frontier VLMs and multimodal embedders (Gemini 3.1 Pro, GPT-family, Qwen3-VL, InternVL, VLM2Vec, Qwen3-VL-Embedding) and find a substantial gap to human consensus; fine-tuning Qwen3-VL-Embedding on their data with three fusion architectures (early / mid / late) narrows the odd-one-out gap from 22.7% to 6.4% below human consensus. Downstream applications include aspect-conditioned nearest-neighbor retrieval on OpenImages (1.7M) and compositional retrieval on WikiArt (800K) where separate aspect queries can be linearly combined.

Key claims

Frontier VLMs, including Gemini 3.1 Pro and Qwen3-VL-Embedding, are ~22.7% behind human consensus on aspect-conditioned odd-one-out judgments over the synthetic test set; aspect-agnostic metrics LPIPS and DreamSim perform near chance since they output a single scalar regardless of the aspect prompt [Fig. 5, §5]. Gemini-3.1-Pro is the strongest baseline; embedding-based Qwen3-VL-Embed is the strongest open-weight baseline and is adopted as the backbone [§5].
Fine-tuning Qwen3-VL-Embedding on the collected data closes the odd-one-out gap from 22.7% to 6.4% below human consensus, and reduces the held-out 2AFC error from a wider baseline gap to 12% below human consensus [Fig. 5, §5].
Three fusion architectures are compared under the same backbone and loss: late fusion (VLM as text-conditioned embedding, cosine similarity between the two image embeddings), mid fusion (LPIPS-style negative-weighted distance between per-layer LLM-decoder activations, with text-conditioned per-channel weights from an MLP), and early fusion (both images + prompt packed into one sequence with a text-conditioned readout register token, satisfying symmetry and identity by construction through attention mask + positional encoding design) [§4.2, Appendix C.2].
In the controlled same-backbone setting, early fusion consistently but modestly outperforms mid and late fusion; late fusion is preferred in practice for retrieval because embeddings can be precomputed and cached [§5].
The dataset pipeline uses text-to-image, LLM, and VLM models as proposers and humans as pruners: FLUX-Reason-6M prompts are diversified by an LLM to produce controlled triplet variations, a VLM proposes candidate aspect lists (including unspecified aspects like pose or object size that emerge from T2I sampling variance), then 5 human annotators per triplet-aspect prune “can’t tell” votes and yield a probability distribution over the three candidates [§3.1].
The training objective is a softmax choice model with cross-entropy against human vote distributions — the same triplet-embedding loss used by Muttenthaler et al., extended to aspect conditioning [§4.1, Eq. 2].
Aspect-conditioned retrieval on OpenImages (1.7M images) yields meaningfully different nearest neighbors for the same query under different aspect conditions; compositional retrieval linearly combines similarity scores across multiple queries conditioned on different aspects, validated on WikiArt (800K images) where axes like subject matter, brushwork, and color palette capture independently meaningful dimensions [§5, Figs. 7-8].

Method

Two datasets sit at the core. The odd-one-out training set contains 24,342 triplets × 257,391 triplet-aspect combinations × 1,044,495 human votes, all generated from FLUX-Reason-6M prompts diversified by an LLM to produce a controlled-but-diverse triplet whose images vary along multiple text-specifiable aspects. A VLM proposes an over-complete aspect list per triplet; 5 human annotators per triplet-aspect prune “can’t tell” aspects and vote for the odd-one-out under each remaining aspect. The out-of-distribution 2AFC set contains 1,127 triplets × 4,771 triplet-aspect × 16,249 votes from four real vision/graphics pipelines (image editing, composite, novel-view synthesis, image-to-3D), used only for evaluation.

The metric is a fine-tuned Qwen3-VL-Embedding with three fusion architectures. Late fusion prompts “Represent the similarity of the image based on ” per image, takes the last-token hidden state as the embedding, and defines similarity as cosine of the two embeddings. Mid fusion feeds each image-prompt pair independently, extracts unit-normalized per-layer image tokens, applies text-conditioned per-channel weights emitted by an MLP over the averaged text tokens, and sums negative weighted distances over selected LLM-decoder layers. Early fusion packs both images + a single prompt into one sequence with two learnable register tokens as text-conditioned readouts; a designed attention mask and positional encoding scheme guarantee d(A,B) = d(B,A) and d(A,A) = 1 by construction. All three train end-to-end with cross-entropy against the human vote distribution under a softmax choice model.

Results

Odd-one-out test set: human consensus ~76%, chance 33%. LPIPS and DreamSim ~35% (near-chance because they are aspect-agnostic). Gemini-3.1-Pro is the strongest baseline at ~53%. Fine-tuned Qwen3-VL-Embed reaches ~70%, a 6.4% gap to human consensus vs the 22.7% gap for the best baseline [Fig. 5].
OOD 2AFC set: fine-tuned model reduces error from the baseline gap to 12% below human consensus, showing generalization from synthetic FLUX training triplets to outputs of external vision/graphics algorithms [§5, Fig. 5].
Qualitative video probe: on a synthesized camera-orbit around a teacup, the “overall” score mirrors the two dominant variation axes (camera angle + sunshine appearing mid-video); prompting for “number of spoons” drops when the spoon disappears behind the cup; prompting for “cup handle position” produces a large drop when the handle rotates from right to left — showing that the aspect prompt selects for the intended visual axis [Fig. 6].
Aspect-conditioned retrieval on OpenImages 1.7M: the same query yields meaningfully different neighbors under different aspects, with an aspect-vs-overall similarity adjustment discouraging degenerate all-overall-similar retrievals [Fig. 7, Appendix B.1].
Compositional retrieval on WikiArt 800K: linear combination of similarity scores across multiple aspect-conditioned queries produces retrievals that jointly satisfy all specified conditions; swapping a single query component produces interpretably different results [Fig. 8].

Why it’s interesting

TPIPS is the most direct realization in the wiki of the “VLM as a text-conditioned perceptual judge” hypothesis. Where UMI-Bench 1.0: An Open and Reproducible Real-World Benchmark for Tabletop Robotic Manipulation with UMI Data-style benchmarks and Rethinking Video Generation Model for the Embodied World (ReVidGen / RBench / RoVid-X)‘s RBench treat the VLM as a judge over a fixed rubric, and where UnifiedReward-Flex (Unified Personalized Reward Model for Vision Generation) dynamically generates the rubric per-prompt before scoring, TPIPS goes further: the rubric is a user-supplied free-form aspect, and the VLM is fine-tuned specifically to produce human-aligned relative comparisons under that aspect. This is a new output shape on the VLM-as-Evaluator enumeration — text-conditioned pairwise similarity — sitting alongside scalar tier, CoT-then-scalar, discrete pick, benchmark-time QA, retrospective synthesis, continuous logit-expectation, and prompt-recovery log-likelihood.

It also validates the Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Ranking embedding backbone as the strongest open-weight starting point for perceptual-judgment fine-tuning, and provides the first filed head-to-head where a late-fusion embedding architecture reaches parity with an early-fusion joint-attention architecture on a task (perceptual similarity) traditionally dominated by the latter. The dataset is likely to matter more than the model: 1M aspect-conditioned human triplet judgments is roughly 10× larger than DreamSim’s release and unlocks aspect-conditioned generative-model evaluation, which the community has been reaching for with hacked-up CLIPScore variants.