Unified Personalized Reward Model for Vision Generation

TL;DR

UnifiedReward-Flex is a VLM-as-judge reward model for text-to-image and text-to-video generation that replaces the conventional one-size-fits-all scoring rubric with per-prompt, context-adaptive hierarchical assessment: for each (prompt, generated content) pair the model first grounds the semantic intent on visual evidence, then dynamically instantiates fine-grained evaluation criteria under both predefined and self-generated high-level dimensions before emitting a reward. Training is a two-stage SFT-then-DPO pipeline — distill structured reasoning traces from closed-source VLMs, then sharpen with DPO on curated preference pairs — and the model is dropped into GRPO post-training of image/video generators. The authors release the 90K SFT dataset, Qwen3-VL- and Qwen35-based checkpoints, and downstream GRPO LoRAs on FLUX.1-dev and FLUX.2-Klein-Base-9B as evidence the reward signal transfers.

Key claims

Existing reward models follow a “one-size-fits-all” paradigm — either fixed scalar scorers like PickScore/CLIP/HPSv3 or Bradley-Terry preference models — that assumes a monolithic preference distribution and is insensitive to content-specific visual cues [§1].
UnifiedReward-Flex couples reward modeling with flexible context-adaptive reasoning: given (prompt, generated content), it interprets semantic intent, grounds on visual evidence, then dynamically constructs a hierarchical assessment by instantiating fine-grained criteria under both predefined and self-generated high-level dimensions [Abstract, §1].
Training pipeline is two-stage: (1) SFT on structured high-quality reasoning traces distilled from advanced closed-source VLMs to bootstrap context-adaptive reasoning behaviors; (2) DPO on curated preference pairs to strengthen reasoning fidelity and discriminative alignment [Abstract].
The reward is plugged into the GRPO framework for image and video synthesis post-training, and is reported to consistently deliver more robust, context-aware reward signals than baselines and to yield substantial improvements across diverse generators [Abstract].
The release ships UnifiedReward-Flex-SFT-90K (training data), checkpoints on Qwen3-VL and Qwen35 backbones, and two downstream LoRAs trained with UnifiedReward-Flex as the GRPO reward on the UniGenBench training set: FLUX.1-dev-UnifiedReward-Flex and FLUX.2-klein-base-9B-UnifiedReward-Flex-lora [project page, HF model cards].
The Flex model sits inside a family of UnifiedReward checkpoints (UnifiedReward, UnifiedReward-2.0, UnifiedReward-Think, UnifiedReward-Edit, UnifiedReward-Flex) and is the only one in the family that supports CoT-style context-adaptive reasoning and the full image/video gen+understand matrix [README compatibility table].

Method

The reward model is a VLM (Qwen3-VL or Qwen35 variants) trained to produce a structured assessment trace rather than a scalar. For each (prompt, image|video) pair it does three things in sequence inside its CoT: (a) interpret the semantic intent of the prompt; (b) ground it on visual evidence in the generated content; (c) dynamically build a hierarchical evaluation by instantiating fine-grained criteria under both predefined high-level dimensions (alignment, coherence, style — inherited from UnifiedReward-2.0) and self-generated high-level dimensions specific to the prompt’s content domain. The final reward is read out at the end of the trace.

Training is SFT + DPO. The SFT corpus is 90K reasoning traces distilled from closed-source VLMs, structured so the student learns the interpret-ground-criteria pattern. DPO then sharpens the discriminator on curated chosen/rejected preference pairs. The trained model is then used as the reward in GRPO post-training of downstream generators — concretely, the authors release FLUX.1-dev and FLUX.2-Klein-9B LoRAs trained against the UniGenBench prompt set with UnifiedReward-Flex as the reward signal. The FLUX.2-Klein-9B LoRA requires ~24-29GB VRAM at inference and is designed to be loaded as a standard diffusion LoRA against the BFL Klein base.

Results

The arxiv body retrieved from the abstract page does not expose the full quantitative tables, but the headline claims from the abstract and project surface are: (1) consistently more robust, context-aware reward signals than existing baselines (PickScore, HPSv3, VideoAlign, LLaVA-Critic, IXC-2.5-Reward, VisionReward, VideoReward, VideoScore, LiFT, and the authors’ own UnifiedReward / UnifiedReward-Think); (2) substantial improvements when used as the GRPO reward across diverse image and video generation models. The compatibility table in the official repo positions UnifiedReward-Flex as the only model in the family that simultaneously covers image+video generation, image+video understanding, pair+point assessment, and CoT reasoning.

The downstream evidence is the FLUX.1-dev and FLUX.2-Klein-Base-9B LoRA releases on HuggingFace — both fine-tuned via GRPO using UnifiedReward-Flex as reward on the UniGenBench training set. The Flex paper and the broader UnifiedReward project also note adoption by other groups: Meta’s Multimodal RewardBench 2 and Transition Matching, NVIDIA’s DiffusionNFT, Tencent Hunyuan’s MixGRPO, ByteDance, and Apple’s UniGen-1.5 cite or build on UnifiedReward.

Why it’s interesting

This is the most explicit attempt yet to fix the failure mode that every aesthetic / reward scorer in ERNIE-Image-Aes: Robust Image Aesthetics Scoring with Balanced Category Generalization documented from a measurement angle: a global rubric biases generation. ERNIE-Image-Aes attacks the problem with a category-balanced training set and a static rubric; UnifiedReward-Flex attacks the same problem by making the rubric itself per-prompt, generated on-the-fly by the VLM judge. For Luma the immediate question is whether a context-adaptive reward beats a per-domain ensemble of static rewards as the GRPO signal for video — the Flex paper claims yes for image/video generators but the body doesn’t expose the head-to-head numbers, so the FLUX.2-Klein-9B LoRA release is the easiest place to A/B in practice. A second thread worth tracking: UnifiedReward-Flex’s “interpret intent → ground → instantiate criteria” pipeline is structurally close to DiffThinker: Towards Generative Multimodal Reasoning with Diffusion Models‘s “MLLM verifier picks among DiffThinker candidates” — same pattern (token-space reasoner verifies pixel-space artifact), but the Flex judge emits a scalar reward instead of a discrete choice, which makes it usable inside GRPO. Third, the design is a foil to HunyuanImage 3.0 Technical Report‘s MixGRPO + multi-reward stack — HunyuanImage 3.0 averages over a fixed bag of reward models; Flex argues a single context-adaptive reward should subsume that bag.