DanceGRPO: Unleashing GRPO on Visual Generation

TL;DR

DanceGRPO is the first unified framework that adapts Group Relative Policy Optimization (GRPO, originally from DeepSeek-R1) to visual-generation post-training, covering both diffusion models and rectified flows across text-to-image, text-to-video, and image-to-video. The key trick is reformulating the deterministic ODE samplers of both paradigms as Stochastic Differential Equations so multiple trajectories per prompt can be drawn, and using group-relative advantages with no KL regularizer. Across Stable Diffusion, FLUX, HunyuanVideo, and SkyReels-I2V, with five reward models (HPS-v2.1, CLIP score, VideoAlign aesthetics / motion, and binary thresholded rewards), DanceGRPO reports up to +181% on motion quality benchmarks and stable training at >10K-prompt scale where prior policy-gradient methods (DDPO, DPOK) destabilized below 100 prompts. This is the recipe that established the “GRPO + shaped reward, no KL” pattern that subsequent video-RL papers cite as default.

Key claims

The sampling processes of both diffusion models and rectified flows can be reformulated as SDEs with controllable stochasticity scale η, enabling the multi-trajectory rollouts GRPO requires; for diffusion the reverse SDE follows [§2.2, Eq. 7], for rectified flows an SDE term is added to the deterministic reverse ODE [§2.2, Eq. 8].
DanceGRPO drops the KL-regularization term from the GRPO objective by default, empirically observing minimal performance difference; this halves memory vs running a frozen reference policy [§2.2].
Samples for the same prompt must share initialization noise to avoid reward hacking — assigning different noise vectors per sample with the same prompt leads to training instability in video generation [§2.2, Fig. 8 in Appendix].
Subsets of timesteps within a denoising trajectory can be omitted without compromising performance — empirically the first 30% of timesteps from noise carry disproportionate weight, but training only on them degrades; a 40% stochastic-timestep dropout is the recommended trade-off [§3.6, Fig. 4(b)].
Multiple reward models should be combined by aggregating advantages rather than rewards directly — different reward models operate on different scales, and HPS-v2.1-only training produces “unnatural (oily) outputs” while adding CLIP score restores realism [§2.2, Fig. 9 in Appendix].
On Stable Diffusion v1.4 (T2I), DanceGRPO raises HPS-v2.1 from 0.239 → 0.365 and CLIP score from 0.363 → 0.421, outperforming DDPO (HPS 0.297), ReFL (0.357), and online-DPO (0.241) [Table 2].
On HunyuanVideo (T2V) with VideoAlign rewards, DanceGRPO achieves +56% relative improvement on visual quality and +181% on motion quality over the base model [§3.3, headline result].
On SkyReels-I2V (I2V), only motion quality is a learnable target (visual fidelity / text-video alignment are constrained by the input image); DanceGRPO reports +91% relative improvement on motion quality, and CFG-training must be enabled to preserve sampling quality during RLHF [§3.4].
The framework supports binary thresholded rewards (above-threshold → 1, below → 0) — thresholding continuous HPS at 0.28 still trains effectively, demonstrating that DeepSeek-R1-style rule-based rewards can transfer to visual generation [§3.2, Fig. 3(a)].
Best-of-N inference scaling: training on the top-k + bottom-k samples from a pool of N (16, 64, 256) per prompt accelerates convergence — the model learns to favor high-reward denoising trajectories, transferring the LLM Best-of-N pattern to diffusion [§3.6, Fig. 4(a)].
Prior RL methods are mutually exclusive on multiple capability axes: DDPO/DPOK lack VideoGen and scalability, ReFL requires differentiable rewards, DPO lacks RL formulation; DanceGRPO is the only filed method covering all axes simultaneously [Table 1].

Method

The sampling chain for both diffusion and rectified flow is formulated as a Markov Decision Process where state is (prompt, timestep, current latent), action is the denoised latent transition, and reward is delivered only at the terminal step from a reward model (HPS-v2.1, CLIP, VideoAlign, etc.). To support GRPO’s group-relative advantage estimation, both paradigms are recast as SDE samplers with a tunable stochasticity scale η — for diffusion this follows the standard score-SDE; for rectified flow a Brownian-motion term is added to the otherwise-deterministic reverse ODE. Both produce stochastic per-step transitions whose log-probability can be evaluated.

For each prompt, G samples are drawn from the policy with shared initialization noise; rewards are scored, advantages are computed as group-normalized rewards Â = (r − μ_group) / σ_group, and the same reward is applied uniformly across all timesteps within a trajectory (reward sparsity is handled by replication, not by per-step decomposition). The policy is updated with a clipped PPO-style ratio objective; KL is dropped by default. To stabilize training at scale, a 40% random timestep dropout is applied during the per-step gradient computation, and when multiple reward models are used their advantages are aggregated (not their raw rewards). For I2V (SkyReels-I2V), CFG-training during the RL phase is required to maintain sampling quality.

The recipe is validated end-to-end across four foundation models (Stable Diffusion v1.4, FLUX.1-dev, HunyuanVideo, SkyReels-I2V), three tasks (T2I, T2V, I2V), and two generative paradigms (diffusion, rectified flow) — a 12-cell coverage matrix where every combination uses the same algorithm. Training runs use 32–64 H800 GPUs depending on task complexity, with always >10K prompts (vs <100 for prior policy-gradient methods).

Results

T2I (Stable Diffusion v1.4): HPS-v2.1 0.239 → 0.365, CLIP 0.363 → 0.421. Beats DDPO (0.297 / 0.381), ReFL (0.357 / 0.418), online-DPO (0.241 / 0.367) [Table 2].
T2V (HunyuanVideo, VideoAlign rewards): Visual Quality 4.51 → 7.03 (+56%), Motion Quality 1.37 → 3.85 (+181%), Text Alignment 1.75 → 1.59 (slight regression; text-alignment dimension was excluded from final training due to instability), VisionReward 0.124 → 0.128 [Table near §3.3].
I2V (SkyReels-I2V): +91% relative on VideoAlign motion quality; visual fidelity and text-alignment are upper-bounded by the input image so are not learnable targets [§3.4].
Binary reward learning: thresholding HPS at 0.28 (and CLIP at 0.39) — DanceGRPO converges on the binary signal despite its sparsity [§3.2, Fig. 3(a)].
Best-of-N curation: training on top-8 + bottom-8 of 16/64/256-sample pools per prompt accelerates Stable Diffusion convergence [§3.6, Fig. 4(a)].
Ablations: noise level η > 0.3 produces noisy outputs after RLHF; reducing η degrades performance — η must be tuned in a narrow band [§3.6, Fig. 4(c)]. Alternative noise decay schedules (e.g. DDPM-style) show no statistically significant difference vs DanceGRPO’s default [§3.6].
Human evaluation: human artists prefer DanceGRPO outputs over base models on 240 T2I prompts (FLUX), 200 T2V prompts (HunyuanVideo), and 200 I2V prompt+image pairs (SkyReels-I2V) [§3.5, Fig. 3(b)].
Stability at scale: trains stably with >10K prompts, vs DDPO/DPOK reported instability beyond ~100 prompts [§3, “stable” framing throughout].

Why it’s interesting

This is the paper that established the GRPO-for-visual-generation recipe currently being inherited everywhere downstream. The “drop the KL term, share initialization noise, aggregate advantages across reward models” trio is now the default in Human detectors are surprisingly powerful reward models (HuDA + Wan 2.1 14B explicitly cites DanceGRPO for the no-KL choice and reports a 73% win-rate over the base model) and is the GRPO half of Reasoning RL‘s “GRPO + shaped reward” pattern. For the Luma post-training phase Pedro flagged: this is the right reference paper to anchor — it gives the SDE reformulation that makes RL on rectified flow work in the first place, plus the reward-aggregation and timestep-dropout details that aren’t obvious from the LLM literature. The contrast with Finite Difference Flow Optimization (FDFO): RL Post-Training for Flow-Based Image Generators is also worth flagging: FDFO argues the per-step MDP framing DanceGRPO uses is itself wasteful for flow models (composing per-step Jacobians distorts the gradient) and gets better results with a no-MDP, image-space finite-difference rule — i.e. DanceGRPO is the strongest GRPO-style baseline, but the post-training design space is not closed.