Data-regularized Reinforcement Learning for Diffusion Models at Scale

TL;DR

DDRL (Data-regularized Diffusion Reinforcement Learning) is NVIDIA Cosmos’s recipe for post-training large-scale diffusion video models with RL while suppressing reward hacking. The diagnosis: reverse-KL regularization in DanceGRPO/FlowGRPO is computed on on-policy rollouts, so as the policy drifts off the data manifold the reference model’s penalty becomes unreliable — and the regularizer fails silently while reward keeps climbing. The fix: swap reverse KL for forward KL anchored to an off-policy data distribution, which by Theorem 3.1 collapses to the standard diffusion loss on real (or synthetic) data, so the RL objective becomes “maximize relative reward + minimize diffusion loss on a fixed dataset.” Across 1M+ H100 hours on Cosmos2.5-2B/14B with VideoAlign and VBench rewards, DDRL is the only method that improves rewards and wins double-blind human voting against the base model (∆-vote = 0 across all four T2V/I2V × VideoAlign/VBench cells, vs. DanceGRPO’s -10.5% and FlowGRPO’s -6.7% on Cosmos2.5-2B T2V).

Key claims

Reverse-KL regularization in DanceGRPO/FlowGRPO is fundamentally on-policy and therefore unreliable: as the policy drifts off the data manifold the reference model is barely trained in those regions, so its KL signal goes to zero even as the generations become visually unacceptable (over-stylized for DanceGRPO, noisy/jittering for FlowGRPO) [§3.1, Fig. 3].
The forward-KL divergence anchored to an off-policy data distribution is equivalent (up to a constant) to the standard diffusion loss on that data, allowing DDRL to be implemented as reward maximization + diffusion loss on a fixed dataset with the same optimum as classical RL [§3.2, Theorem 3.1, Eq. 9].
The dataset used for the diffusion-loss term can be either the real high-quality dataset that trained the reference model, or synthetic samples drawn from the reference model — both work; the T2I experiments with SD3.5-Medium use purely synthetic data [§4.4, Table 4].
On Cosmos2.5-2B with VideoAlign T2V rewards, DDRL is the only method with a non-negative ∆-vote against itself (0% by construction); DanceGRPO loses to DDRL by 10.5%, FlowGRPO by 6.7%, and the base model by 22.9% [Table 1].
DDRL achieves a Pareto improvement across the three VideoAlign sub-scores (visual quality, motion quality, text alignment), while DanceGRPO improves average reward by sacrificing text alignment — a direct numerical fingerprint of reward hacking [§4.2, Fig. 4].
DDRL drops the reference and old-policy networks from memory: the diffusion-loss data is collected upfront (real or synthetic), so neither a reference model nor an old-policy model is needed during training, unlike FlowGRPO/DanceGRPO [§3.3, Memory & Efficiency].
The number of network evaluations (NFEs) per data point can be reduced from N×K (rollouts × timesteps) to N+K by sampling timesteps independently and optimizing the diffusion loss only once, matching DanceGRPO’s NFE cost (no KL) and halving FlowGRPO’s [Table 3, §3.3].
DDRL can replace the standard SFT→RL pipeline with a single stage: applied directly to a vanilla pretrained checkpoint with a high-quality dataset, it reaches comparable rewards (and equivalent diffusion loss, 0.119 vs 0.121) to a 20K-iteration SFT-then-RL pipeline [§4.3, Fig. 5].
DDRL omits classifier-free guidance during the diffusion-loss computation and drops the condition with some probability (following diffusion training standard), in contrast to FlowGRPO which uses CFG and DanceGRPO which avoids it whenever possible [§3.3, Timesteps & Guidance].
On T2I (SD3.5-Medium, OCR reward, LoRA finetuning), DDRL achieves comparable OCR reward (0.823) to DanceGRPO (0.846) and FlowGRPO (0.845) while preserving the base model’s PickScore (0.865) and ImageReward (1.14), where DanceGRPO regresses to 0.837 PickScore / 0.80 ImageReward — i.e. the OOD-reward gap is the reward-hacking gap [Table 4].
A practical reward-hacking detector: if the diffusion loss of a post-trained checkpoint increases by more than ~10%, the outputs are mostly hacked; steep reward increases and sharp reward-stdev decreases are also indicative [§4.2].

Method

DDRL replaces the reverse-KL term of GRPO with a forward-KL term anchored to an off-policy data distribution μ_ψ. Writing q(x_t | x) for the forward noising process, the objective is E[A(x) · log p_θ(x_t | x)] − β · L_diffusion(p_θ, μ_ψ), where A(x) is the group-relative advantage (mean-centered, optionally stdev-normalized, as in GRPO) and L_diffusion is the standard ε-prediction diffusion loss on data sampled from μ_ψ. Theorem 3.1 shows the optimum of this objective matches the classical RL optimum p_θ*(x) ∝ p_ref(x) · exp(r(x)/τ) whenever μ_ψ is the reference model — i.e. the data regularization is mathematically equivalent to anchoring on the reference, but computed off-policy.

In practice DDRL needs only (a) the policy p_θ, (b) the reward model, and (c) a fixed dataset for the diffusion-loss term — no reference model in memory, no old-policy model. For each training iteration the algorithm draws G rollouts from p_θ, scores them, computes group-relative advantages, accumulates the reward-maximization gradient over half the denoising steps (every other timestep, following DanceGRPO’s “subsets of timesteps” finding), and adds a diffusion-loss gradient on a freshly-sampled timestep and datapoint from μ_ψ. The ablations in Table 3 show that sampling one diffusion-loss timestep per rollout (rather than K) is sufficient, dropping the per-rollout NFE count to DanceGRPO levels.

The Cosmos2.5 experiments use 256 H100s for the 2B model and 1024 H100s for the 14B model, with rollout size 16 and a 1:1 T2V/I2V data mixture. Reward models are hosted on a separate 512-GPU server and called asynchronously. Total budget: ~1M H100 hours plus ~10K double-blind human votes across fifteen researchers.

Results

Cosmos2.5-2B T2V (VideoAlign): DDRL reward 0.604 vs DanceGRPO 0.715 / FlowGRPO 0.408 / base 0.408. ∆-vote vs. DDRL: DanceGRPO −10.5%, FlowGRPO −6.7%, base −22.9% [Table 1]. DanceGRPO’s higher reward but worse human preference is the reward-hacking signature.
Cosmos2.5-14B T2V (VideoAlign): DDRL reward 0.555, ∆-vote vs. base −8.4%, vs. DanceGRPO −4.9%, vs. FlowGRPO −6.7% [Table 1]. Smaller margin because all methods used a reduced learning rate (3e-6) to suppress hacking in baselines.
VBench T2V/I2V on both Cosmos2.5 sizes: DDRL wins on every cell against every baseline, with smaller absolute reward gaps (VBench scores cluster in 0.81–0.84) but consistent ∆-vote ≥ 3% [Table 1].
Pareto sub-score improvement: on VideoAlign sub-scores DDRL improves all three (visual, motion, text-alignment) simultaneously; DanceGRPO trades text-alignment for the other two [§4.2, Fig. 4]. This is the cleanest signal in the paper that “average reward goes up” is the wrong metric.
Longer training is safe for DDRL: doubling iterations from 128 → 256 increases VideoAlign T2V reward from 0.604 → 0.627 without degrading human preference, while DanceGRPO and FlowGRPO degrade visibly past 128 iterations [Table 3, Fig. 3].
NFE efficiency: with the “sample one timestep per rollout” simplification, DDRL hits ~N+K NFEs per rollout (matching DanceGRPO without KL), vs. FlowGRPO’s ~2N+K [Table 3 — exact column labels are the NFE counts].
SFT-replacement: applying DDRL directly to a vanilla pretrained Cosmos2.5 checkpoint reaches diffusion loss 0.119 vs. 0.121 for the 20K-iteration SFT-then-RL baseline, with comparable rewards [§4.3, Fig. 5]. The 20K SFT iterations are eliminated, though the dominant cost (on-policy rollouts) is unchanged.
T2I (SD3.5-Medium, OCR reward, LoRA r=128): DDRL OCR 0.823, ClipScore 0.320, PickScore 0.865, ImageReward 1.14 — all four metrics preserved at or above base model. DanceGRPO: OCR 0.846 (+0.28 vs. base) but PickScore 0.837 (−0.028) and ImageReward 0.80 (−0.33), and loses to DDRL by 53.0% in human voting [Table 4]. Reward-hacking magnitude correlates with how badly the off-reward metrics drop.
Hyperparameter sweep over baselines: >15% of total compute spent sweeping FlowGRPO’s β and training iterations to make sure baselines were tuned charitably; β across two magnitudes does not prevent reward hacking [§4.2].

Why it’s interesting

DDRL is the first filed paper that proposes a principled diagnosis of why DanceGRPO and FlowGRPO reward-hack — both lineages of the dominant GRPO-for-diffusion recipe summarized in Reasoning RL and instantiated in DanceGRPO: Unleashing GRPO on Visual Generation — and replaces them with a forward-KL formulation that reduces to plain diffusion-loss regularization. Three things make this notable for Luma’s video post-training agenda: (a) the diagnosis directly contradicts the “drop KL by default” claim in DanceGRPO: Unleashing GRPO on Visual Generation §2.2 — DDRL is adding an explicit regularizer back, just one that works off-policy via diffusion loss; (b) the SFT-replacement result (§4.3) argues post-training stages should be unified rather than chained, complementing the FDFO line in Finite Difference Flow Optimization (FDFO): RL Post-Training for Flow-Based Image Generators (which independently argues the standard per-step MDP for flow-RL is wrong) — both papers reject the inherited GRPO framing but go in different directions (DDRL keeps the MDP, fixes the regularizer; FDFO drops the MDP entirely); (c) it’s the first filed result on the wiki for RL post-training on a Cosmos-scale world-foundation model, and the asynchronous-reward-server / 1024-GPU recipe is directly relevant to scaling video RL.

The “diffusion loss going up by 10%+ ≈ hacked” detection heuristic in §4.2 is also a cheap monitor any team running diffusion RL should add — it’s an offline metric on the same data that doesn’t require the human voting that bottlenecked this paper’s evaluation.