Finite Difference Flow Optimization (FDFO): RL Post-Training for Flow-Based Image Generators

TL;DR

FDFO is an online RL post-training algorithm for flow-matching image generators that avoids the denoising-MDP framework entirely. It generates a pair of images from the same starting noise (with sampling stochasticity making them diverge), takes the image-space difference weighted by reward difference as a finite-difference reward gradient, and applies that single image-space update uniformly to the velocity prediction at every step of the sampling chain — rather than backpropagating through the chain or treating each step as an MDP action. The author reports FDFO converges faster than Flow-GRPO on Stable Diffusion 3.5 Medium, reaches higher reward, produces fewer artifacts, and — once post-trained — removes the need for CFG at inference. A separate experiment trains a LoRA on-policy from ~3,200 human pairwise preferences via a simple web GUI, in near real-time.

Key claims

The denoising-MDP framing used by DDPO / DRaFT / Flow-GRPO conflates credit assignment across denoising steps with reward maximization; under it each per-step stochastic perturbation is reinforced when its trajectory wins, so “much of the update is noise that pushes the flow in reward-neutral directions” [Method, “Currently, the most popular algorithm…”].
Backpropagating an image-space reward gradient all the way up the sampling chain via the chain of per-step Jacobians is feasible when the reward is differentiable but is poorly conditioned: the composed Jacobians distort the gradient’s frequency content, so the update direction at early steps is dominated by a narrow eigen-direction [Method, “It’s worth analyzing what the backpropagation is doing”].
A copied image-space gradient — applied uniformly to all earlier steps without backpropagation through the sampling chain — is a valid ascent direction whenever the flow’s per-step Jacobian is approximately positive-semidefinite (i.e. nearly straight, which is what flow-matching training tries to produce in the first place) [Method, “Flow matching models are tuned to produce flows with minimal curvature”].
Empirically, the copied (biased) gradient outperforms the exact backpropagated gradient — uniform application of the true reward gradient is comparable to the full method, while backpropagating that same gradient through the trajectory is “significantly worse” [Method, reported finding that the copied gradient beats the exact backpropagated gradient; Fig. 8 rows I vs J].
For non-differentiable rewards, generate a pair of samples from the same noise with sampling-time stochasticity making them diverge; the image difference weighted by reward difference is a finite-difference estimate of the reward gradient along that axis. Apply it uniformly across the flow as in the differentiable case [Method, “FDFO Algorithm”].
After FDFO post-training of Stable Diffusion 3.5 Medium, the model produces “high-quality, well-aligned images on its own, without CFG”; re-enabling CFG mainly reduces diversity and introduces the high-contrast “AI look” with debatable benefit; FDFO reaches similar reward levels regardless of inference CFG scale [Results, “Before post-training, the base model’s output quality without CFG is poor”].
Flow-GRPO is reported to periodically produce grid-like artifacts during extended training and consistent style drift across prompts; FDFO is reported not to under matched training budgets, which the author attributes to lower-variance updates [Results, “Flow-GRPO periodically produces grid-like artifacts”].
A direct human-feedback variant is built around a web GUI: pairs of images per prompt, user picks the preferred one, ~30 s of training per batch, served back in the next round; ~3,200 pairwise labels were sufficient to impart a personal aesthetic on the LoRA [Human-feedback section, web-GUI implementation note].
Updating all noise levels uniformly is “important” in the human-feedback regime — low-frequency structure (composition, layout) and high-frequency detail (textures, sharpness) are corrected simultaneously rather than the high-frequency end dominating [Human-feedback section, “One practical finding from this experiment”].
Independent quantitative support: OneIG-Bench is invoked alongside reward curves to corroborate the claim that FDFO improves prompt alignment and image quality versus Flow-GRPO at matched compute [Results, “Independent metrics support these findings”].

Method

The setup is a standard text-to-image flow-matching model (Stable Diffusion 3.5 Medium) post-trained against a scalar reward. The default reward is a VLM’s P("Yes" | "Does this image match [CAPTION]?") likelihood; the algorithm is reward-agnostic.

Mechanically, FDFO does three things differently from DDPO / DRaFT / Flow-GRPO:

Reward signal. Instead of treating each denoising step as an MDP action and computing a per-step policy-gradient log-ratio, FDFO computes a single image-space update direction per pair of trajectories. For differentiable rewards, this is the image-space reward gradient (one backward pass through the VLM, not through the sampling chain). For non-differentiable rewards, it is a finite difference: sample two images from the same initial noise with sampling-time noise injection making them diverge, then use (reward_A − reward_B) · (image_A − image_B) as the direction.
Credit assignment. That image-space direction is copied verbatim to the loss for the velocity prediction at every step of the sampling chain — no Jacobian composition, no MDP advantages, no per-step reweighting. The argument: for sufficiently straight flows (the regime flow-matching is trained to produce), the per-step Jacobian is approximately PSD, so the image-space gradient is within 90° of the true per-step gradient at every step and is therefore a valid ascent direction. Backpropagating through the chain in principle gives a better-aligned direction at each step, but the composition collapses the gradient’s frequency content and conditions it badly.
CFG handling. No explicit CFG term in the RL objective. The post-trained model is reported to no longer need CFG at inference — a side-effect of the reward sharpening the conditional distribution onto the well-aligned mode that CFG was approximating.

The pair-sampling step gives FDFO a natural mapping onto on-policy pairwise human preferences: two images per prompt, user picks one, the difference image weighted by +1 / −1 is the update direction. A small web GUI is reported to run this loop in near real-time (~30 s training per batch between rounds), with ~3,200 labels imparting a coherent personal style on a LoRA over SD3.5 Medium.

Results

Sample efficiency vs. Flow-GRPO. Reported faster convergence and higher final reward at matched training compute on SD3.5 Medium with the VLM “Yes”-token reward [Results, charts referenced but unembedded].
Artifacts. Flow-GRPO is reported to periodically produce grid-like artifacts and consistent style drift; FDFO does not under the same training budgets [Results, “The quality of training also differs”].
CFG ablation. Pre-FDFO, no-CFG quality is poor and CFG is standard practice. Post-FDFO, no-CFG quality matches or exceeds with-CFG; re-enabling CFG mainly reduces diversity and introduces a high-contrast “AI” look. FDFO reaches similar reward levels regardless of inference CFG scale [Results, “Before post-training, the base model’s output quality without CFG is poor”].
Independent metric. OneIG-Bench results corroborate the reward-curve story [Results, “Independent metrics support these findings”]. Specific deltas are not extracted in this filing.
Human-feedback variant. ~3,200 pairwise labels over an interactive loop are reported to impart a coherent personal aesthetic on a LoRA over SD3.5 Medium, with both compositional and textural changes accumulating across rounds [Human-feedback section].
Ablation on uniform-vs-backprop credit assignment. Uniform application of the true reward gradient to all steps is comparable to FDFO’s full pair-sampling recipe; backpropagating that same gradient through the trajectory is significantly worse [Method, Fig. 8 rows I vs J].

Why it’s interesting

FDFO is a deliberately flow-native RL post-training recipe — it sidesteps the denoising-MDP scaffolding that DDPO and Flow-GRPO impose, and instead exploits flow-matching’s straightness property to justify a much simpler update rule. This puts it in direct dialogue with two filed threads. First, on the RL side: Reasoning RL documents GRPO-family RL spreading from LLMs to VLMs (vlm-gym: RL gym for vision language models (JAX)) and to video diffusion policies (Human detectors are surprisingly powerful reward models‘s HuDA on Wan 2.1 14B) — both of which keep the MDP framing and shape the reward; FDFO takes the orthogonal route of replacing the MDP framing on image diffusion. Second, on the CFG side: FDFO’s “post-training removes the need for CFG” claim is a third route to the same destination as Continuous Adversarial Flow Models (CAFM’s adversarial criterion cures generalization-miscalibration) and Random noise augmentation during diffusion training eliminates the need for CFG (Ostris’s noise-augmentation cures noise-miscalibration). The three are independent recipes converging on “CFG is patching something that can be patched upstream”; whether they compose is now a real open question. The human-feedback variant is also interesting on its own — pairwise on-policy with ~3,200 labels and a 30 s training-step is a very small-data regime by RLHF standards, and the same primitive (pair sampling + finite-difference direction) is what makes it work.