Cycle Consistency as Reward: Learning Image-Text Alignment without Human Preferences

TL;DR

CycleReward learns an image-text alignment reward without any human or GPT-4V preference labels by using cycle consistency as the supervisory signal: given a candidate caption for an image, map the caption back to image space with a T2I model and score how close the reconstruction is to the original; symmetrically for T2I. The authors release CyclePrefDB, 866K cycle-consistency-derived preference pairs spanning 11 I2T and 4 T2I decoders, and train a BLIP-backed Bradley-Terry reward model on it. CycleReward beats human-preference-trained reward models (HPSv2, PickScore, ImageReward) on detailed-captioning agreement and is competitive on T2I, and using CyclePrefDB for DPO improves Qwen-VL-Chat and Stable Diffusion 1.5 across a wide range of vision-language benchmarks.

Key claims

Cycle consistency — sim(x, G(F(x))) for an image-to-text encoder F and text-to-image decoder G, and symmetrically sim(y, F(G(y))) — can serve as a self-supervised proxy for image-text alignment preferences without human or VLM-judge supervision [§3.1].
Computing cycle consistency on-the-fly with large pretrained models (as in Image2Text2Image or DDPO) is slow and non-differentiable; distilling the signal into a small BLIP-backed reward model recovers both speed and differentiability while improving alignment quality [§2, §5.1].
CyclePrefDB contains 866K comparison pairs (398K I2T from 11 image-to-text decoders × 7.6K DCI images; 468K T2I from 4 text-to-image decoders × 12 candidates per prompt), with average caption length 55-57 tokens — substantially denser than the 19-36-token captions in HPDv2 / Pick-a-Pic v2 / ImageRewardDB [Table 1].
Cycle consistency preferences agree with human preferences at 65.6% average across RLHF-V, POVID, HPDv2, Pick-a-Pic v2, and ImageRewardDB for the joint-trained CycleReward-Combo — higher than GPT-4o-as-judge (48%) when averaged across all five [Table 2].
On DetailCaps-4870 (detailed-captioning pairwise accuracy), CycleReward-Combo scores 60.5%, outperforming VQAScore-11B (50.2%, ~24× larger) and the human-preference-trained HPSv2 (54.3%), PickScore (51.0%), and ImageReward (50.7%) [Table 3].
DPO finetuning of Qwen-VL-Chat on CyclePrefDB-I2T raises DecapBench from 26.5 → 30.6 and improves LLaVA-WD detailed-captioning while preserving general VQA (MMHalBench, MMEP, MMEC) — outperforming DPO on VLFeedback (GPT-4V labels) on detailed captioning despite no human or VLM-judge supervision [Table 6].
Diffusion-DPO of Stable Diffusion 1.5 on CyclePrefDB-T2I beats Diffusion-DPO on Pick-A-Pic across most T2I-CompBench compositional categories (Spatial 16.6 vs 14.6, Color 42.4 vs 39.1, Complex 37.8 vs 34.7, Texture 46.7 vs 40.7) and is roughly tied on long-prompt PartiPrompts [Table 7].
Ablations identify three load-bearing choices: DreamSim (perceptual) image-similarity outperforms LPIPS / raw-CLIP-cosine for image-side cycle consistency; Bradley-Terry loss beats MSE regression of the raw cycle score (58.0 → 41.9 on DetailCaps-4870); stronger I2T and T2I decoders (InternVL2-26B, Stable Diffusion 3) produce better preference labels and therefore better learned rewards [Tables 4-5, §5.3].
The DCI-train training set is small (7.6K images) but its dense captions are the load-bearing input: training on a 1K subset drops DetailCaps-4870 pairwise accuracy from 58.0 → 52.9 and GenAI-Bench from 53.5 → 44.4 [§5.3].

Method

CycleReward derives preferences from one of two scoring functions: S(t|i) = sim(i, G(t)) for evaluating a candidate caption t against an image i using a fixed T2I decoder G, and S(i|t) = sim(t, F(i)) for evaluating a candidate image against a caption using a fixed I2T decoder F. The chosen similarities are DreamSim for images (trained to match human perceptual judgments) and SBERT for text; both significantly outperform CLIP cosine in the ablation. Pairwise preferences come straight out of these scalar scores — given two candidates with S(t_a) > S(t_b), t_a is preferred.

Dataset construction uses 7.6K DCI images. For I2T, eleven captioners spanning BLIP2, LLaVA-1.5 / 1.6 (7B, 13B, 34B), LLaVA-OneVision (0.5B, 7B), and InternVL2 (2B, 8B, 26B, 40B) generate captions at the 77-token T2I limit; the fixed backward map is Stable Diffusion 3. For T2I, four diffusion models (SD1.5, SDXL, SD3, FLUX-Schnell) generate 12 candidates per prompt (3 seeds × 4 models); the fixed backward map is LLaVA-1.5-13B. The reward model is a BLIP-style ViT-L/16 + text encoder + 5-layer MLP, trained with Bradley-Terry loss on either dataset alone (CycleReward-I2T, CycleReward-T2I) or jointly with α=1.0 weighting (CycleReward-Combo). The “raw cycle consistency” baseline computes S directly at evaluation time without ever fitting a reward model — it is what Image2Text2Image [Bahng et al. 2024] uses as a metric — and CycleReward beats it across the board, isolating the contribution of distillation.

Results

Agreement with human preferences (Table 2): CycleReward-Combo 65.6% avg; CycleReward-T2I 65.5%; CycleReward-I2T 64.3%; raw cycle consistency 58.9%; GPT-4o 48.0%. The gap between raw-on-the-fly and the learned reward model justifies distillation by itself.
Detailed-captioning alignment (Table 3): CycleReward-Combo 60.5%, CycleReward-I2T 58.0% on DetailCaps-4870, vs HPSv2 54.3% / PickScore 51.0% / ImageReward 50.7% / VQAScore-11B 50.2% / CLIPScore 51.7%.
T2I alignment (GenAI-Bench): CycleReward variants 53.5-55.5% vs HPSv2 56.1% / PickScore 57.1% / ImageReward 56.7% / VQAScore-11B 64.1%. CycleReward is competitive with human-preference reward models but does not match large-VLM VQAScore here.
Best-of-N captioning: BoN-250 over LLaVA-1.5-13B captions selected by CycleReward yields the largest LLaVA-WD and DeCapBench overall-score gains; analysis on DeCapBench non-hallucination vs comprehensiveness shows CycleReward weights detail more than VQAScore / ImageReward, which over-prefer accurate-but-short captions [§5.2, Fig. 3].
DPO on Qwen-VL-Chat [Table 6]: DecapBench 26.5 → 30.6 (CycleReward-DPO) vs 28.0 (VLFeedback-DPO); LLaVA-WD 61.7 → 70.0 vs 69.2; general VQA broadly preserved or improved.
Diffusion-DPO on Stable Diffusion 1.5 [Table 7]: Spatial 11.5 → 16.6, Color 37.0 → 42.4, Complex 34.5 → 37.8, Texture 40.4 → 46.7 — beats Pick-A-Pic-DPO on 5 of 6 T2I-CompBench categories with smaller training set (468K vs 851K pairs).
Ablation crossovers: MSE on raw cycle scores → 41.9 / 40.6 (vs 58.0 / 53.5 for BT loss). 1K-image training → 52.9 / 44.4. No filtering → 57.3 / 51.9. Stronger I2T decoder (InternVL2-26B vs LLaVA-1.5-13B for the backward map in T2I training) → 57.2 vs 51.7 on DetailCaps-4870.

Why it’s interesting

CycleReward is a new shape on the VLM-as-Evaluator page: prior filed entries split into static-rubric scoring (ERNIE-Image-Aes: Robust Image Aesthetics Scoring with Balanced Category Generalization), per-prompt CoT-then-scalar (Unified Personalized Reward Model for Vision Generation), discrete-pick verifier (DiffThinker: Towards Generative Multimodal Reasoning with Diffusion Models), and benchmark-time LMM-judge with human-written questions. CycleReward adds a fifth — fully self-supervised reward distillation from generator-pair cycle consistency, with no human, GPT-4V, or even hand-crafted rubric input. The thesis is structurally close to CHAI: Building a Precise Video Language with Human–AI Oversight (CHAI also trains a domain-specialized reward model on detailed-caption pairs) but moves further out: CHAI replaces frontier-VLM judges with human critiques, CycleReward replaces them with another generator running in reverse. The 866K-pair CyclePrefDB is also a clean Synthetic Training Data datapoint — the cycle-consistency score is the verification gate, and Bradley-Terry-over-cycle-scores is the load-bearing detail that distinguishes “works” from MSE-on-raw-scores (“doesn’t, by 16-17 points”). For Luma, the obvious test is whether G(F(video)) on a video model + a robust video-similarity metric (e.g. DreamSim-Video, VBench dimensions, or a learned video-clip embedding) gives a usable signal for prompt-adherence in Ray3 post-training; the original-prompt-vs-recaption variant Yunong proposes is a single-direction degenerate case of this same loop, which CycleReward’s “raw cycle consistency” baseline (Table 2) already shows is weaker than the distilled reward model.