Qwen-Image-2.0-RL Technical Report

TL;DR

Qwen-Image-2.0-RL is the post-training recipe layered on top of the Qwen-Image-2.0 unified text-to-image + editing diffusion model. Two specialized GRPO policies — one for T2I, one for image editing — are each trained against fine-tuned VLM-based composite reward models that emit pointwise CoT scores, then merged into a single deployable student via on-policy distillation with trajectory-level velocity matching. The recipe ships three stabilizers that earlier RLHF-for-diffusion attempts lacked: a hybrid classifier-free guidance strategy that preserves pretrained knowledge during RL, intra-group reward-range filtering for prompt curation, and per-category reward weight calibration. End-to-end the post-training adds +2.61 overall on Qwen-Image-Bench and lifts Elo by +78 (T2I arena) and +93 (edit arena) over the Qwen-Image-2.0 base.

Key claims

• Two task-specific composite reward models are constructed by fine-tuning VLMs with a pointwise scoring paradigm and CoT reasoning: T2I covers alignment, aesthetics, and portrait fidelity; editing covers instruction-following accuracy and face identity preservation [Abstract].
• A scalable GRPO-based RL framework drives both T2I and editing post-training, with three explicit stabilizers — hybrid CFG to preserve pretrained knowledge, intra-group reward-range filtering for prompt curation, and per-category reward weight calibration [Abstract].
• The two task-specialized RL policies (T2I and edit) are merged into a single student via on-policy distillation with trajectory-level velocity matching, treating the two specialists as teachers and consolidating them in a final training stage [Abstract].
• Qwen-Image-2.0-RL reaches 57.84 overall on Qwen-Image-Bench, a +2.61 lift over the Qwen-Image-2.0 base [Abstract].
• Arena Elo lifts: +78 in T2I arena (1115 → 1193) and +93 in image edit arena (1256 → 1349) [Abstract].

Method

The pipeline has three stages. (1) Reward modeling: separate composite reward models are built per task by SFT-ing a VLM under a pointwise scoring paradigm with chain-of-thought reasoning — the VLM emits a CoT trace and a scalar per dimension (alignment / aesthetics / portrait fidelity for T2I; instruction-following / face-ID preservation for edit). The pointwise CoT-then-scalar shape is the same family as UnifiedReward-Flex (Unified Personalized Reward Model for Vision Generation) and inherits the per-prompt context-adaptive rubric pattern. (2) GRPO post-training: two specialist policies are trained against the two reward stacks. The three stabilizers — hybrid CFG (sampling configuration that keeps the pretrained two-pass behaviour as a knowledge prior during RL rollouts), intra-group reward-range filtering (drop prompt groups whose reward variance is too low to be informative), and per-category reward weight calibration (rebalance the scalar weights on each reward dimension so no single head dominates) — directly target the reward-variance-collapse and reward-hacking failure modes that DDRL (Data-regularized Reinforcement Learning for Diffusion Models at Scale) and Stable-Layers (Stable-Layers: Fine-Tuning Image Layer Decomposition Models with VLM-Scored Reinforcement Learning) flagged at video and layered-image scale respectively. (3) On-policy distillation merge: the two RL specialists are distilled into a single student with trajectory-level velocity matching — at each denoising step, the student’s predicted velocity is regressed against the appropriate specialist teacher’s velocity along the on-policy student trajectory. This merges the two GRPO policies into one inference-time model.

Results

On Qwen-Image-Bench (the Qwen-side benchmark-time T2I judge — see Qwen-Image-Bench: From Generation to Creation in Text-to-Image Evaluation), the post-trained model scores 57.84 overall, +2.61 over the base Qwen-Image-2.0 [Abstract]. On AI Arena Elo, T2I goes from 1115 → 1193 (+78) and image editing from 1256 → 1349 (+93) [Abstract]. The base Qwen-Image-2.0 was already reported #1 on AI Arena at release (Qwen-Image-2.0); RL post-training is the source of the additional gains. The Abstract attributes the lifts to gains across aesthetic quality, prompt adherence, and editing accuracy, consistent with the per-dimension reward decomposition.

Why it’s interesting

This is the first filed paper that publishes the full RL post-training recipe for the Qwen-Image lineage, closing the gap flagged in Qwen-Image-2.0 (where the 2.0 release omitted technical details on RLHF / DPO / GRPO). The three GRPO stabilizers — hybrid CFG, intra-group reward-range filtering, per-category weight calibration — directly attack the same failure modes that Stable-Layers: Fine-Tuning Image Layer Decomposition Models with VLM-Scored Reinforcement Learning surfaced in Flow-GRPO on Qwen-Image-Layered (pointwise CoT scoring collapsing within-group variance) and that Data-regularized Reinforcement Learning for Diffusion Models at Scale surfaced in DDRL on Cosmos video (reward hacking under unconstrained RL). The on-policy distillation merge step is the most interesting structural choice: it treats the T2I and edit specialists as two teachers and uses trajectory-level velocity matching to consolidate them — the diffusion-side analogue of the LLM-side on-policy distillation pattern (On-Policy Distillation) applied to merge multi-task RL policies rather than to imitate a single larger teacher. Together with Uniworld-V2: Reinforce Image Editing with Diffusion Negative-aware Finetuning and MLLM Implicit Feedback (Uniworld-V2 / Edit-R1, DiffusionNFT + MLLM implicit feedback on the edit side) and Pref-GRPO: Pairwise Preference Reward-based GRPO for Stable Text-to-Image Reinforcement Learning (pairwise-preference GRPO), this rounds out a small but growing set of recipes for putting GRPO-trained diffusion image models into production.

See also

• Qwen-Image-2.0 — the base model this paper post-trains; explicitly notes the RL recipe was not public at the time.
• Qwen-Image — Image foundation model with text rendering and editing (QwenLM/Qwen-Image) — Qwen-Image release lineage (Qwen-Image 1.0 → Edit → 2.0 → 2.0-RL).
• Qwen-Image-Bench: From Generation to Creation in Text-to-Image Evaluation — Qwen-Image-Bench, the benchmark used to report the +2.61 overall lift.
• Unified Personalized Reward Model for Vision Generation — UnifiedReward-Flex, same pointwise-CoT-then-scalar VLM-reward shape plugged into GRPO; sibling recipe.
• Stable-Layers: Fine-Tuning Image Layer Decomposition Models with VLM-Scored Reinforcement Learning — Stable-Layers, prior datapoint on pointwise-CoT-reward-collapse under Flow-GRPO and the pairwise-grid fix; directly relevant to the intra-group filtering choice here.
• Data-regularized Reinforcement Learning for Diffusion Models at Scale — DDRL (NVIDIA Cosmos), the video-scale analogue with its own anti-reward-hacking regularization.
• Uniworld-V2: Reinforce Image Editing with Diffusion Negative-aware Finetuning and MLLM Implicit Feedback — Uniworld-V2 / Edit-R1, contemporary edit-side GRPO recipe with DiffusionNFT.
• Pref-GRPO: Pairwise Preference Reward-based GRPO for Stable Text-to-Image Reinforcement Learning — Pref-GRPO, the pairwise-preference alternative to pointwise CoT scoring.
• On-Policy Distillation — the LLM-side reference for the on-policy distillation step used here to merge the two RL specialists.
• Reasoning RL — GRPO recipe family that this paper is the latest image-generation instance of.
• VLM-as-Evaluator — VLM-as-reward-model paradigm this paper instantiates.
• Classifier-Free Guidance variants — hybrid CFG stabilizer at training time fits into this cluster’s “CFG as a training-time problem” theme.
• Diffusion Distillation — on-policy distillation merge stage.