RewardDance: Reward Scaling in Visual Generation

TL;DR

RewardDance is ByteDance Seed’s framework for scaling visual reward models along two axes: parameter count (1B → 26B InternVL backbones) and context (task instructions + reference images + CoT reasoning). The core trick is reformulating the reward score as the VLM’s predicted probability of a “yes” token in a pairwise “is image1 better than image2?” judgment — eliminating the regression head that breaks the next-token-prediction alignment of standard VLM-based scorers. Applied to RL fine-tuning (ReFL) and inference-time search-over-paths on FLUX.1-dev, Seedream-3.0, and Seedance-1.0, scaling the RM monotonically improves T2I/T2V/I2V quality and — critically — preserves high reward variance throughout training, suppressing the reward hacking / mode collapse that plagues smaller regressive RMs.

Key claims

The “yes-token probability” generative reward formulation natively aligns reward modeling with VLMs’ autoregressive next-token-prediction mechanism, removing the architectural mismatch introduced by appending a regression head trained with Bradley-Terry loss [§3.2.1, Eq. 2].
Scaling the reward model from 1B to 26B parameters improves OOD preference accuracy from 69.10% to 80.90%, while in-domain accuracy is non-monotonic (1B 64.70% / 2B 69.36% / 4B 65.37% / 8B 74.92% / 26B 78.44%) — the paper argues OOD generalization is the more predictive metric for downstream RL gains than ID accuracy [Table 2].
RL fine-tuning Seedream-3.0-SFT with a 26B RM raises the Bench-240 alignment score from 74.1 baseline to 84.8; FLUX.1-dev improves from 67.0 to 73.6; test-time scaling alone (search-over-paths with the 26B RM as verifier) takes Seedream-3.0-SFT from 74.1 to 80.5 [Table 2].
Video generation shows the same scaling: Seedance-1.0-SFT GSB improvement grows from +28% (1B RM) to +49% (26B RM) on T2V, and +29% to +47% on I2V — a consistent monotonic scaling effect across modalities [Table 3].
Large RMs sustain high reward variance throughout ReFL training while small RMs converge to narrow error bars early; the authors interpret the wide sustained variance as evidence the policy is still exploring under the 26B RM, and read this as resistance to reward hacking [§4.4, Fig. 5–6].
Seedream-3.0 + RewardDance reaches 0.79 overall on GenEval (+0.10 over the SFT-only baseline at 0.69); FLUX.1-dev + RewardDance reaches 0.75 (+0.09 over 0.66 baseline) [Table 4].
Seedream-3.0 + RewardDance scores 0.848 overall on Bench-240, beating Imagen 3 (0.79), Ideogram (0.77), Luma (0.77), Qwen-Image (0.77), Midjourney V6.1 (0.63), DALLE-3 (0.67), and FLUX1.1 (0.67) [Table 5].
Seedance-1.0 + RewardDance posts the top T2V Video-Text Alignment of 1.66 on SeedVideoBench-1.0, ahead of Veo-3.0 (1.63) and Kling 2.1 Master (1.57); ties Kling 2.1 at 1.65 on I2V [Table 6].
The framework also defines a pointwise generative variant for inference-time search-over-paths: same “yes-token probability” readout but without reference images, trained with Bradley-Terry plus a small cross-entropy auxiliary loss to ground the yes/no labels [§3.2.3, §3.3.2].
ReFL training uses Best-of-N reference selection: N candidates are sampled from the SFT model, pairwise-compared by RewardDance, and the top-ranked image becomes the reference for subsequent RL steps [§3.3.1].
CoT data is constructed in two formats — “decision-then-reasoning” and “reasoning-then-decision” — distilled from SEED-VL 1.5 as the teacher; the decision-then-reasoning format is preferred at RL time so the “yes” probability can be read directly from the first output token [§3.2.1, Fig. 4].

Method

The architectural change is small but consequential: replace the regression head with the VLM’s native LM head, and define the reward as r = P_VLM("yes" | image1, image2, prompt, instruction), i.e. the probability the next token is “yes” given a pairwise comparison prompt. This makes reward modeling a literal next-token-prediction task aligned with VLM pretraining. RewardDance is built on the InternVL family at five sizes (1B, 2B, 4B, 8B, 26B) and trained on pairwise preference data augmented with task-specific instructions and chain-of-thought rationales (distilled from SEED-VL 1.5). Training data includes both ordering variants (yes/no first vs reasoning first) so the model learns both behaviors; at inference the decision-first variant is used to keep the readout cheap.

For RL fine-tuning, RewardDance plugs into the ReFL algorithm with a Best-of-N reference selection: for each prompt, N candidates are sampled from the SFT model, RewardDance does pairwise comparisons among them, and the top-ranked image becomes the reference for subsequent ReFL steps; the diffusion model is then optimized to maximize P("yes") on (candidate, reference) pairs. For inference-time scaling, the framework instead uses search-over-paths: N noise trajectories are extended through ODE sampling, and at each branching step a lighter pointwise generative variant (no reference image, evaluates a single candidate against task instructions) prunes to the most promising trajectories. Evaluation runs on Bench-240 (T2I, 240 prompts), SeedVideoBench-1.0 (T2V/I2V, 300 prompts), and GenEval (compositional T2I), with in-house human alignment scoring and an industry-comparable Video-Text Alignment 0/1/2 rubric.

Results

T2I scaling on Seedream-3.0-SFT (74.1 baseline): 1B → 26B RM in RL fine-tuning lifts Bench-240 alignment from 74.1 to 84.8 (+10.7); test-time scaling alone lifts to 80.5 [Table 2].
T2I scaling on FLUX.1-dev (67.0 baseline): 26B RM lifts to 73.6 (+6.6) on the same benchmark [Table 2].
T2V scaling on Seedance-1.0-SFT: GSB win-rate over SFT improves +28% → +49% as RM grows 1B → 26B [Table 3].
I2V scaling on Seedance-1.0-SFT: GSB win-rate improves +29% → +47% over the same RM range [Table 3].
GenEval (compositional T2I): Seedream-3.0 0.69 → 0.79 (+0.10); FLUX.1-dev 0.66 → 0.75 (+0.09); the FLUX baseline at 0.66 sits below SD-v3 at 0.74, so the +0.09 RewardDance lift moves it above SD-v3 [Table 4].
Bench-240 head-to-head: Seedream-3.0 + RewardDance 0.848 overall vs the strongest closed-source baseline Imagen 3 at 0.79, Qwen-Image 0.77, Ideogram 0.77, Luma 0.77 [Table 5].
SeedVideoBench-1.0 T2V: Seedance-1.0 + RewardDance 1.66 vs Veo-3.0 1.63, Kling 2.1 Master 1.57, Wan 2.1 1.49, Sora 1.37 [Table 6].
SeedVideoBench-1.0 I2V: Seedance-1.0 + RewardDance 1.65, tied with Kling 2.1 Master 1.65, ahead of Veo-3.0 1.59, Kling 2.0 Master 1.57, Wan 2.1 1.36, RunwayGen 4 1.37 [Table 6].
RM accuracy decoupling: ID accuracy is non-monotonic across the 1B–26B sweep; OOD accuracy rises monotonically (69.10% → 80.90%) — the paper argues OOD accuracy is the practically actionable metric [Table 2].
Reward dynamics: 26B RM maintains substantially wider reward variance bands deep into ReFL training compared to 1B/2B RMs, which converge to narrow bands early; the paper reads this as “exploration capacity preserved” rather than “policy collapsing onto reward-hacking shortcuts” [§4.4, Figs. 5–6].

Why it’s interesting

RewardDance is the first systematic scaling study for visual reward models, and its conclusions reshape several of this wiki’s open questions. For VLM-as-Evaluator, it confirms the “structured CoT trace then read out a scalar” shape (Unified Personalized Reward Model for Vision Generation‘s UnifiedReward-Flex direction) is the right composition with RL, and provides the missing scaling-law evidence: bigger generative-VLM judges with richer context get monotonically better at guiding the generator. For Reasoning RL and the GRPO-on-visual-generation lineage anchored by DanceGRPO: Unleashing GRPO on Visual Generation, it directly addresses the open question of whether the no-KL on-policy recipe causes reward hacking under longer training — the answer here is “depends on RM size: small RMs do, 26B RMs don’t, the variance bands tell you which regime you’re in”. The author overlap with DanceGRPO (Jie Wu, Zeyue Xue, Wei Liu, Weilin Huang) is not a coincidence; this is the reward-model half of the same ByteDance Seed alignment stack, and Seedance-1.0 + RewardDance is exactly the RL recipe behind the production Seedance numbers in Seedance 1.0: Exploring the Boundaries of Video Generation Models.