HY-SOAR: Self-Correction for Optimal Alignment and Refinement in Diffusion Models

TL;DR

HY-SOAR (Self-correction for Optimal Alignment and Refinement) is a Tencent HY-team post-training framework for rectified-flow diffusion models that targets exposure bias — the mismatch between the clean (ground-truth) noisy states the denoiser sees in training and the model-induced (off-trajectory) noisy states it sees at inference. The recipe samples an on-trajectory noisy state, performs one stop-gradient CFG rollout with the current model to a clean image, re-noises that off-trajectory clean image back to the same noise endpoint, and trains the denoiser with an analytical flow-matching correction target on the resulting on-policy state. No reward model, no preference labels, no negatives — the supervision is dense (per-timestep) and the off-trajectory distribution co-evolves with the model. On SD3.5-Medium at 512² with cfg=4.5, SOAR lifts GenEval from 0.70 (SFT) to 0.78 (+11% relative) and simultaneously improves every DrawBench preference and quality metric, beating Flow-GRPO on its own reward objectives (Aesthetic 5.94 vs 5.87, ClipScore 0.300 vs 0.296) while training reward-free.

Key claims

Exposure bias — the gap between training noisy states (drawn from the forward process on clean data) and inference noisy states (drawn from the model’s own denoising rollout) — is the structural cause of compounding denoising failures in rectified-flow models; SOAR is framed as a direct fix [Project page, “Directly addresses the mismatch…”].
A single CFG rollout from an on-trajectory noisy state with stop-gradient produces an off-trajectory clean image that is on-policy with respect to the current model; the training distribution therefore co-evolves with the model rather than staying fixed at the data distribution [Project page, “Off-trajectory states are produced by the current model’s own rollout”].
The off-trajectory clean image is re-noised toward the same noise endpoint as the original on-trajectory pair, keeping the auxiliary state near the original transport ray and yielding an analytical correction target compatible with the flow-matching objective [Project page, “Re-noising uses the same noise endpoint as the base flow-matching pair”].
The SOAR loss extends standard flow-matching supervision and is positioned as a stronger first stage that can replace SFT, while remaining compatible with downstream RL alignment (Flow-GRPO and friends layered on top) [Project page, “The SOAR loss extends the standard flow-matching objective and can replace SFT”].
No reward model, no preference labels, no negative samples; supervision is per-timestep and dense, avoiding the terminal-reward credit-assignment problem of RL-style post-training [Project page, “Requires no reward model, preference labels, or negative samples”].
SD3.5-Medium at 10k iterations, 512² / cfg=4.5: SFT lifts GenEval 0.63 → 0.70 / OCR 0.59 → 0.64 / ImgRwd 0.85 → 1.04; SOAR lifts the same base to GenEval 0.78 / OCR 0.67 / ImgRwd 1.09, with PickScore 22.86, ClipScore 0.295, HPSv2.1 0.289, Aesthetic 5.46 [Main results table].
SOAR’s final scores on Flow-GRPO’s own reward objectives beat Flow-GRPO at matched training: Aesthetic 5.94 vs SFT 5.74 / Flow-GRPO 5.87; ClipScore 0.300 vs SFT 0.297 / Flow-GRPO 0.296 — despite SOAR being reward-free [Project page, “SOAR’s final scores not only surpass SFT but also outperform Flow-GRPO, which explicitly uses these metrics as its reward signal”].
Qualitative comparisons highlight text-rendering and compositional fidelity on design/poster prompts, and accurate layout, typography, and visual hierarchy on web-UI / graphic-design prompts [Project page, qualitative galleries].

Method

The setting is a pretrained rectified-flow / flow-matching image generator (SD3.5-Medium in the headline experiments). Standard SFT and standard RL alignment (Flow-GRPO, DDPO, DRaFT) are the two reference baselines.

The SOAR step proceeds as follows:

Sample an on-trajectory pair (x_0, ε) from data and Gaussian noise, and pick a timestep t. Form x_t on the standard flow-matching transport ray between x_0 and ε — this is the same state SFT would see.
CFG rollout with stop-gradient. With the current denoiser frozen (no gradient), perform a CFG-guided denoising rollout from x_t to produce a clean image x̂_0. Because the rollout uses the current model with CFG, x̂_0 is off-trajectory but on-policy.
Re-noise to the same endpoint. Form a new noisy state x̂_t on the flow-matching ray between x̂_0 and the same ε as in step 1, at the same t (or a sampled timestep). This keeps the auxiliary state near the original transport ray, so the analytical flow-matching target is well-defined.
Analytical correction target. Train the denoiser at x̂_t with the rectified-flow velocity target derived from (x̂_0, ε, t). The supervision is dense (any t), per-timestep, and uses no reward model.

Because step 2 stop-grads the rollout, gradients flow only through step 4’s flow-matching loss — the recipe avoids backpropagation through the sampling chain entirely. Because step 3 re-uses the same noise endpoint, the auxiliary state stays close to the model’s actual transport ray rather than drifting onto an arbitrary off-trajectory region.

SOAR is positioned as a first-stage post-training recipe — applied to a pretrained / SFT’d checkpoint, replacing or improving on SFT before any RL or preference-based alignment stage.

Results

All reported numbers are SD3.5-Medium at 10k iterations, 512² / cfg=4.5, evaluated on DrawBench (preference and aesthetic metrics) and GenEval / OCR test sets, following the Flow-GRPO protocol.

SD3.5-Medium base → +SFT (10k) → +SOAR (10k): GenEval 0.63 → 0.70 → 0.78 (+11% relative over SFT); OCR 0.59 → 0.64 → 0.67; PickScore 22.34 → 22.71 → 22.86; ClipScore 0.285 → 0.295 → 0.295; HPSv2.1 0.279 → 0.284 → 0.289; Aesthetic 5.36 → 5.35 → 5.46; ImgRwd 0.85 → 1.04 → 1.09 [Main results table].
Comparison to larger baselines: SOAR-tuned SD3.5-M’s GenEval 0.78 exceeds FLUX.1-Dev (0.66) and SD3.5-L at 1024² (0.71); OCR 0.67 vs SD3.5-L 0.68 / FLUX.1-Dev 0.59 [Main results table].
Beats Flow-GRPO on Flow-GRPO’s own reward objectives: DrawBench Aesthetic 5.94 (SOAR) vs 5.87 (Flow-GRPO) vs 5.74 (SFT); ClipScore 0.300 (SOAR) vs 0.296 (Flow-GRPO) vs 0.297 (SFT) [Project page, head-to-head section].
Qualitative: stronger text rendering and compositional fidelity on design / poster prompts; accurate layout and typography on web-UI / graphic-design generation; comparisons span aesthetic-quality, text-image alignment, and graphic-design prompt families [Project page, qualitative galleries].
Reward-free throughout: no reward model trained, no preference labels, no negative samples [Project page, claims and abstract].

The arxiv version is not yet released; numbers and ablations beyond the headline table are not available from the project page alone.

Why it’s interesting

SOAR is a third, structurally distinct recipe for post-training flow-matching image generators, complementary to the two recipes already on the wiki. (1) Distillation-side: Diffusion Distillation catalogs train/inference-gap fixes — TMD’s flow-head rollout inside DMD (Transition Matching Distillation for Fast Video Generation), DiagDistill’s Diagonal Forcing (Streaming Autoregressive Video Generation via Diagonal Distillation), and Causal Forcing’s three-stage AR-teacher fix (Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generation) all attack the same exposure-bias-of-the-student problem that SOAR attacks in the teacher / generator itself. SOAR’s “rollout-then-re-noise-to-the-same-endpoint” is the generator-side analogue of TMD’s “roll out the flow head during DMD” — both make supervision occur over states the model actually visits at inference. (2) RL-side: Finite Difference Flow Optimization (FDFO): RL Post-Training for Flow-Based Image Generators (FDFO) and TDM-R1: Reinforcing Few-Step Diffusion Models with Non-Differentiable Reward (TDM-R1) are the wiki’s two flow-native RL recipes, both reward-based; SOAR explicitly drops the reward and replaces it with the model’s own CFG rollout as the “correction signal.” The natural composition is SOAR-as-first-stage → FDFO/TDM-R1 as second-stage, which the project page suggests but does not demonstrate. (3) CFG-side: like Continuous Adversarial Flow Models (CAFM) and Random noise augmentation during diffusion training eliminates the need for CFG (Ostris’s noise-augmentation result), SOAR uses CFG as a training-time signal rather than treating it as inference-time inference machinery — but unlike those two, it does not claim CFG-removal, only that CFG-rollout-with-stop-grad is a useful training-time operator. The most striking single number is SOAR beating Flow-GRPO on Flow-GRPO’s own reward metrics while training reward-free; if the arxiv version replicates this on FLUX-class / Qwen-Image-class generators, the practical case for shaped-reward RL on flow models gets weaker.