Reinforcing Diffusion Models by Direct Group Preference Optimization

TL;DR

DGPO (Direct Group Preference Optimization) is an online RL recipe for post-training diffusion models that drops the policy-gradient framework entirely. The motivation is mechanical: GRPO-style methods (DanceGRPO, FlowGRPO) need a stochastic policy to define a tractable per-step log-probability, which forces the use of SDE-based samplers — and the model-agnostic Gaussian noise those samplers inject is what makes training slow. DGPO instead supervises the diffusion model directly on the group-level preference signal (which sample beat which inside a group of rollouts), bypassing per-step log-probabilities and unlocking efficient deterministic ODE samplers. Reported result: ≈20× faster training than existing SOTA RL methods for diffusion, with superior in-domain and out-of-domain reward scores. The recipe sits alongside DiffusionNFT (the paper Jiaming flagged as similar in spirit) as one of two contemporary attempts to escape the SDE/GRPO trap by reformulating RL on diffusion as a supervised-learning problem over preference data rather than a policy-gradient problem over sampling trajectories.

Key claims

GRPO formally requires a stochastic policy to define per-step log-probabilities; this forces SDE-based samplers, whose model-agnostic Gaussian noise injection is the dominant source of slow convergence in current diffusion-RL recipes [§1, §2].
DGPO dispenses with the policy-gradient framework entirely and learns directly from group-level preferences, using the relative information of samples within a group as the supervision signal rather than a normalized scalar advantage [Abstract, §3].
Because DGPO does not need per-step log-probabilities, training can run on deterministic ODE samplers (the cheapest known diffusion sampler family) rather than the more expensive SDE alternatives required by GRPO/FlowGRPO [Abstract, §3].
DGPO trains roughly 20× faster than existing SOTA diffusion-RL methods and achieves superior performance on both in-domain and out-of-domain reward metrics [Abstract].
Code is released at github.com/Luo-Yihong/DGPO [Abstract].

Method

DGPO post-trains a diffusion model by sampling a group of G outputs per prompt with an ODE sampler, scoring them with a reward model, and then converting the group’s scores into a preference structure (which sample beat which) rather than a vector of advantages. The training objective then directly optimizes the diffusion model to make the preferred samples more likely under the forward (denoising) process — analogous in spirit to how DPO replaces RLHF’s policy-gradient inner loop with a direct supervised objective on preference pairs, but extended to group-level (not pairwise) preferences and adapted to the flow-matching / diffusion training loss rather than a log-ratio over autoregressive tokens.

This is the same overall move that DiffusionNFT (Chen et al., Sept 2025) makes from a different angle: rather than discretize the reverse sampling process into an MDP and run GRPO over it (FlowGRPO/DanceGRPO), reformulate the RL problem so the diffusion model can be trained on clean outputs with a modified loss. DiffusionNFT calls its mechanism Negative-aware FineTuning on the forward process (contrasting positive vs negative generations to define an implicit policy-improvement direction); DGPO calls its mechanism Direct Group Preference Optimization (group-level preferences instead of a contrastive signal). Both share the conclusion that the per-step MDP framing is the wrong abstraction for diffusion RL.

Results

≈20× wall-clock training speedup over existing SOTA diffusion-RL methods at matched performance [Abstract].
Superior in-domain reward scores after training (vs. GRPO-family baselines) [Abstract].
Superior out-of-domain reward generalization — the model does not appear to overfit the training reward model in the way pointwise-RM GRPO does [Abstract].

(Full per-benchmark numbers and ablations are in the paper; the abstract gates the headline claims and the full PDF was not retrievable at filing time.)

Why it’s interesting

DGPO and DiffusionNFT (the paper Jiaming flagged as the spiritual ancestor) are two near-simultaneous proposals that the post-DanceGRPO recipe — GRPO over SDE-discretized diffusion sampling — is structurally wasteful, and that the fix is to reformulate diffusion RL as a supervised objective over preference data on clean outputs rather than a policy-gradient objective over noisy trajectories. This is the same critique Finite Difference Flow Optimization (FDFO): RL Post-Training for Flow-Based Image Generators makes from yet another angle (FDFO argues the per-step Jacobian composition in the MDP framing distorts the gradient and proposes a finite-difference rule in image space). Three independent attempts to escape the GRPO/SDE trap in <6 months is the strongest signal the wiki has seen that the recipe established by DanceGRPO: Unleashing GRPO on Visual Generation is being actively superseded.

For Reasoning RL this directly extends the page’s “GRPO is the de facto recipe” claim with a counterpoint: the de facto recipe may not survive 2026 in the diffusion sub-domain. For Classifier-Free Guidance variants the connection is via SDE-vs-ODE — DGPO unlocking ODE samplers for RL is the same kind of “stop paying the CFG/SDE tax at training time” move that the CFG-elimination thread (SoFlow, Ostris) makes at inference time. The natural ablation nobody has run yet is DGPO + Pref-GRPO: Pairwise Preference Reward-based GRPO for Stable Text-to-Image Reinforcement Learning‘s pairwise win-rate reward shape — both attack reward hacking, from orthogonal angles (reward shape vs. removing policy gradient).