Pref-GRPO: Pairwise Preference Reward-based GRPO for Stable Text-to-Image Reinforcement Learning

TL;DR

Pref-GRPO replaces the pointwise reward-model score in GRPO-based T2I post-training with a pairwise win-rate computed inside each generated group. The motivation is mechanical: when a pointwise RM assigns near-identical scores across a group (a very common regime once the policy has converged somewhat), the group-normalized advantage (r − μ)/σ blows up tiny score differences into large gradients — the paper calls this illusory advantage and identifies it as the root cause of reward hacking in DanceGRPO-style recipes. Replacing the reward signal with each image’s pairwise win-rate (proportion of intra-group comparisons it won, judged by UnifiedReward-Think) amplifies real variance, suppresses noise amplification, and yields more stable training. On FLUX.1-dev the recipe lifts UniGenBench overall from 61.30 (base) and 63.62 (UnifiedReward pointwise GRPO) to 69.46, with the biggest gains on Logical Reasoning (+12) and Text Rendering (+13). The paper also ships UniGenBench, an MLLM-judged 600-prompt T2I benchmark with 10 primary and 27 sub-dimensions.

Key claims

Reward hacking in pointwise-RM GRPO is driven by illusory advantage: because pointwise RMs assign overly similar scores to comparable images within a group, the group standard deviation σ collapses, and Â = (r − μ)/σ disproportionately amplifies small score differences into large gradients [§3.2, Eq. 8].
Pairwise win-rate as the reward signal — for each image in a group of size G, enumerate all G(G−1)/2 pairs, ask a pairwise preference RM which is preferred, define the reward as the proportion of pairwise comparisons won — produces inherently amplified reward variance with high-quality samples approaching win-rate 1 and low-quality approaching 0 [§3.3, Eq. 9].
The pairwise reformulation provides three properties simultaneously: amplified reward variance, robustness to noise/bias in the RM (relative ranking is less sensitive than raw scores), and closer alignment with human pairwise judgment of comparable images [§3.3].
On FLUX.1-dev, Pref-GRPO lifts UniGenBench Overall from 61.30 to 69.46 (+8.16), beating UnifiedReward pointwise GRPO at 63.62 by +5.84 absolute; gains are concentrated on Logical Reasoning (30.91 → 44.09) and Text Rendering (32.18 → 47.13) [Table 2].
Pointwise-RM baselines exhibit visible reward hacking in qualitative output: HPS-optimized images become oversaturated, UnifiedReward-optimized images become unnaturally dark; combining HPS + CLIP reduces but does not eliminate the artifact [§5.2, Fig. 6].
Reward scores rise rapidly during pointwise-RM training while inspected image quality degrades; Pref-GRPO’s score growth is more gradual but image quality improves consistently [§5.2, Fig. 2].
Pref-GRPO transfers to out-of-domain benchmarks: GenEval rises from 62.92 (FLUX.1-dev base) to 70.53; T2I-CompBench rises from 48.17 to 51.85 [Table comparing Semantic Consistency vs Image Quality].
Image-quality metrics independent of the training RM also improve: UnifiedReward score 3.04 → 3.26, PickScore 22.42 → 23.02, ImageReward 1.27 → 1.44, Aesthetic 6.13 → 6.52 [same table].
UniGenBench, the accompanying benchmark, covers 600 prompts across 5 themes × 20 subthemes with 10 primary and 27 sub-evaluation dimensions, with Gemini-2.5-Pro as the MLLM judge producing per-testpoint binary scores plus textual rationales [§4.1, §4.2].
On UniGenBench, GPT-4o (92.77) and Imagen-4.0-Ultra (91.54) lead all models; Qwen-Image (78.81) is the strongest open-source model; Logical Reasoning and Text Rendering remain unsaturated across both open and closed-source models [Table 3].

Method

Pref-GRPO inherits the SDE reformulation of flow matching from DanceGRPO so that multi-trajectory rollouts are possible on a rectified-flow policy (FLUX.1-dev). The change is only in the reward step. For each prompt, G images are sampled with shared initialization noise; instead of feeding each through a pointwise RM and standardizing the scalar outputs, the framework enumerates all pairs and queries a pairwise preference RM — UnifiedReward-Think, which is a VLM trained to emit a “preferred image” decision in a pairwise comparison prompt — for each pair. Each image’s reward is its observed win rate across its G−1 comparisons. These win-rates are then standardized into group-relative advantages and plugged into the standard GRPO clipped-ratio objective, with KL omitted following DanceGRPO. No other RL-stage hyperparameter changes are made.

UniGenBench’s construction is fully MLLM-automated. For each prompt, Gemini-2.5-Pro samples a (theme, subject category, testpoint set) tuple uniformly at random, generates a natural-language prompt satisfying those semantic constraints, and emits structured per-testpoint descriptions explaining how each testpoint is realized in the prompt. At evaluation time, Gemini-2.5-Pro receives the generated image, the prompt, and the testpoint descriptions, and emits binary per-testpoint scores plus textual rationales; sub-dimension scores aggregate the binaries, and primary-dimension scores aggregate the sub-dimensions.

Results

Reward-design head-to-head on FLUX.1-dev (UniGenBench Overall): base 61.30 → HPS 58.77 (slightly worse) → HPS+CLIP 61.81 → UnifiedReward pointwise 63.62 → Pref-GRPO 69.46 [Table 2].
Per-dimension lift over UnifiedReward pointwise on UniGenBench: Style 86.10 → 88.40 (+2.30), World Knowledge 89.72 → 90.35 (+0.63), Attribute 71.55 → 75.00 (+3.45), Action 63.69 → 69.77 (+6.08), Relations 70.42 → 76.52 (+6.10), Logical Reasoning 32.05 → 44.09 (+12.04), Grammar 62.43 → 63.27 (+0.84), Compound 52.32 → 62.43 (+10.11), Layout 73.51 → 77.61 (+4.10), Text 34.44 → 47.13 (+12.69) [Table 2].
OOD semantic-consistency benchmarks (FLUX.1-dev): T2I-CompBench 48.17 → 51.85; GenEval 62.92 → 70.53 [comparison table].
OOD image-quality metrics independent of the training RM: UnifiedReward 3.04 → 3.26, PickScore 22.42 → 23.02, ImageReward 1.27 → 1.44, Aesthetic 6.13 → 6.52 [comparison table].
Qualitative reward-hacking signature: HPS pushes outputs toward oversaturation, UnifiedReward pushes toward darkness; HPS+CLIP combination reduces both pathologies but neither fully eliminates them; Pref-GRPO preserves natural image statistics while improving semantics [§5.2, Fig. 6].
Closed-source vs open-source ceiling on UniGenBench: GPT-4o 92.77, Imagen-4.0-Ultra 91.54, FLUX-Kontext-Max 80.00 / Seedream-3.0 78.95 (closed-source top); open-source: Qwen-Image 78.81, HiDream 71.81, FLUX.1-dev + Pref-GRPO 69.46 — making Pref-GRPO a competitive open-source recipe though not the absolute frontier [Table 3].
Hard-dimension unsaturation: even GPT-4o scores 84.97 on Logical Reasoning and 89.24 on Text Rendering — Pref-GRPO’s +12-point lifts on these dimensions are the largest gains it claims because they are also where headroom is largest [Table 3].

Why it’s interesting

Pref-GRPO completes a triangle with DanceGRPO: Unleashing GRPO on Visual Generation and RewardDance: Reward Scaling in Visual Generation that the wiki was missing a third corner of. DanceGRPO established the no-KL group-relative recipe but treated reward hacking as a phenomenon to manage with reward-aggregation and Best-of-N curation; RewardDance argued that scaling the reward model (1B → 26B) is what suppresses hacking because larger RMs preserve reward variance deeper into training; Pref-GRPO argues the reward-function shape itself — pairwise win-rate vs pointwise score — is the underlying lever, and that a relatively modest preference RM gives Pref-GRPO the same variance-preserving property that RewardDance gets from sheer parameter count. The author overlap is suggestive (Yibin Wang of UnifiedReward also authored UnifiedReward-Think used here), so this is the reward-shape half of the same Shanghai-AI-Lab / Tencent alignment stack rather than an external alternative. For the wiki’s Reasoning RL page, this also directly answers one of the listed open questions: yes, the no-KL DanceGRPO recipe does cause reward hacking under longer training, and the fix is in the reward signal, not the policy update.