Flow Matching Policy Gradients

TL;DR

FPO (Flow Policy Optimization) is an on-policy RL algorithm that swaps the Gaussian-likelihood-ratio in PPO for an advantage-weighted conditional flow matching ratio, letting practitioners train diffusion/flow-based policies from scratch with PPO-style stability. Unlike prior diffusion-RL methods (DDPO, DPPO, Flow-GRPO) that reformulate the denoising chain itself as an MDP — multiplying the horizon by 10–50× and binding training to a specific sampler — FPO treats sampling as a black box at rollout time and uses CFM loss as a likelihood proxy. On 10 MuJoCo Playground tasks, FPO beats both Gaussian PPO and DPPO in 8/10; on humanoid mocap tracking, it dominates Gaussian PPO in the under-conditioned regime (root-only or root+hands targets), where the multimodal expressivity of flow policies matters most.

Key claims

FPO’s ratio is exp(-β · (L_CFM(θ) - L_CFM(θ_old))) substituted into the PPO-clip objective in place of the Gaussian likelihood ratio, where L_CFM is the conditional flow matching loss; this avoids the need for exact flow-model likelihoods entirely [§3.2, Eq. 5–7].
The substitution is justified by Kingma & Gao’s result that the weighted denoising loss equals the negative ELBO up to a constant for the standard diffusion schedule, so the ratio of ELBOs decomposes into the true likelihood ratio times an inverse KL-gap correction — maximizing it improves both [§3.3, Eq. 11–12, §3.4, Eq. 15–16].
The ratio estimate is biased upward (Jensen’s inequality on exp) when computed from few (t, ε) samples, but the gradient is directionally unbiased; empirically FPO works with n=1 samples per update and improves with more [§3.4, Eq. 17–20].
FPO is agnostic to the sampler used at rollout and inference time — deterministic or stochastic, first- or higher-order, any number of integration steps — unlike denoising-MDP methods which are bound to their specific sampling chain [§3.5].
FPO learns multimodal action distributions in the GridWorld saddle-point setting, where two equally-good goals exist; a Gaussian PPO baseline collapses to the nearest goal [§4.1, Fig. 1].
On MuJoCo Playground (10 DM Control Suite tasks, 5 seeds each, 60M env steps), FPO with 8 (t,ε) samples and ε-MSE weighting reaches average reward 759.3, beating DPPO (652.5) and tuned Gaussian PPO (667.9) [§4.2, Table 1; Fig. 3].
In humanoid mocap tracking on AMASS via Puffer-PHC: with full-joint conditioning FPO ≈ Gaussian PPO; with under-conditioning (root-only or root+hands targets) FPO substantially outperforms — a regime where prior work needed teacher-student distillation [§4.3, Table 2, Fig. 4].
Hyperparameter sensitivity: clipping ε behaves like in standard PPO (ε=0.1 best in their sweep, ε=0.2 collapses); ε-MSE weighting (predicting noise) outperforms v-MSE (predicting velocity), hypothesized to be because noise scale is action-scale-invariant [§4.2, Table 1].
DPPO-style methods inflate the credit-assignment horizon by treating initial noise as an environment observation and each denoising step as a Gaussian action — increasing the effective MDP dimensionality and forbidding deterministic samplers — both of which FPO avoids by construction [§3.5].

Method

The setup is standard PPO with two surgical changes. (1) The policy is replaced by a flow model v_θ(x_t, s, t) that, given state s, integrates Gaussian noise into an action via the usual denoising sequence — same primitive as image/video flow matching, just operating on action vectors. (2) The PPO likelihood ratio π_θ(a|s) / π_θ_old(a|s) is replaced by exp(-β · ΔL_CFM), where ΔL_CFM is the per-sample difference in conditional flow matching loss under the current vs. old parameters, averaged over n Monte-Carlo draws of (t, ε). Everything else — GAE advantage estimation, the PPO-clip surrogate, actor-critic interleaving — stays. The mathematical justification is that for the standard diffusion schedule, the weighted denoising loss equals the negative ELBO up to an additive constant (Kingma & Gao 2023), so the CFM-loss-ratio equals an ELBO-ratio which decomposes into the true likelihood ratio scaled by an inverse-KL-gap correction; maximizing it pushes the policy toward high-advantage actions and tightens the variational gap.

The contrast with prior diffusion-RL (DDPO, DPPO, Flow-GRPO) is that those methods reframe the denoising chain itself as an MDP: each denoising step is treated as a Gaussian sub-action with its own log-likelihood, and the trajectory is augmented with the per-step sampling noise as observations. This (a) multiplies the credit-assignment horizon by the number of denoising steps (typically 10–50), (b) ties training to whatever stochastic sampler was used, and (c) blows up state dimensionality. FPO sidesteps all of this by not opening up the sampling chain at all — it sees only the final action and asks “what’s the CFM loss for this (s, a) pair under both policies?”

Results

MuJoCo Playground (10 DMC tasks, 5 seeds, 60M env steps): FPO 759.3 ± 45.3 avg reward vs DPPO 652.5 ± 83.7 vs Gaussian PPO 667.9 ± 66.0; wins on 8/10 tasks individually [Table 1, Fig. 3].
GridWorld saddle-point: FPO captures bimodal action distribution at the saddle point and produces diverse trajectories from the same start state; Gaussian PPO collapses to the nearest goal [Fig. 1].
Humanoid mocap tracking on AMASS (Puffer-PHC / Isaac Gym, 24-joint SMPL humanoid):
- Full-joint conditioning: FPO ≈ Gaussian PPO across success rate, alive duration, MPJPE.
- Root+hands (VR-controller-like) conditioning: FPO outperforms substantially.
- Root-only conditioning: FPO outperforms even more, while Gaussian PPO struggles to learn viable walking [Table 2].
FPO ablations on MuJoCo Playground: n=8 (8 samples per update) reward 759.3; n=4 731.2; n=1 691.6 — more samples help but n=1 already beats Gaussian PPO. ε-MSE (predict noise) 759.3 > v-MSE (predict velocity) 664.6. Clipping ε=0.1 best, ε=0.2 collapses to 526.4 [Table 1].
Terrain transfer: FPO trained with terrain randomization enables the humanoid to traverse rough terrain — sim-to-real signal, no real-robot results in the paper [§4.3, Fig. 4c].

Why it’s interesting

FPO sits in the same family as Finite Difference Flow Optimization (FDFO): RL Post-Training for Flow-Based Image Generators (same first author, follow-up work for image generators) — both deliberately refuse the denoising-MDP framing that DDPO/DPPO/Flow-GRPO impose on diffusion-RL. FPO does it for action spaces with a CFM-loss-ratio in PPO; FDFO does it for image spaces with a finite-difference reward gradient applied uniformly across denoising steps. The common move is “treat the sampler as a black box and update the flow with one image/action-space signal,” and it’s now winning in both control (this paper) and post-training (FDFO). On the Reasoning RL page, this is the first filed reasoning-RL recipe targeting continuous control / robotics with a flow-based policy — every other filed GRPO-family paper there targets LLMs, VLMs, video diffusion, or multi-agent orchestration; FPO extends the family to a domain (robot control) where Gaussian policies have been the unchallenged default for years. The most striking practical result is humanoid tracking under sparse (root or root+hands) conditioning — a setting where prior work needed teacher-student distillation and FPO solves it in one stage by exploiting the multimodal capacity of flow policies.