VINE: Taming Generative Control Policies for Reinforcement Learning

TL;DR

VINE is a training-time end-to-end value-gradient RL recipe for flow-matching control policies that keeps all of iterative multi-step denoising, high expressiveness, and backprop through the entire denoising chain — the three ingredients prior work (FPO, RECAP, QGF, DiPOD) had to give up at least one of. The key diagnosis is that the instability blamed on iterative generation actually comes from the vanilla flow-matching sampling strategy inherited from behavior cloning, which is brittle when the loss becomes a value gradient rather than a BC loss. VINE replaces the single-trajectory sampler with a re-interpolation step at every denoising step, reconstructing a fresh interpolation state that provides a stable differentiable path for value-gradient propagation while remaining compatible with the original denoising process. Reported to backprop through all ten denoising steps stably, beat state-of-the-art RL baselines on OGBench offline-RL, and transfer to a real-world robotic manipulation task.

Key claims

Prior belief that value-gradient RL instability on flow-matching policies is caused by the iterative multi-step denoising itself is wrong; the instability is caused by the vanilla sampling strategy (originally designed for behavior cloning) becoming brittle when the training loss is a value gradient rather than a supervised BC target [Abstract].
Existing workarounds — sacrifice iterative generation, sacrifice expressiveness, or sacrifice end-to-end value-gradient optimization — are all responses to the wrong diagnosis and therefore leave capability on the table [Abstract].
VINE reconstructs a new interpolation state at every denoising step instead of following a single flow trajectory, creating a stable differentiable path along which the value gradient can propagate through the entire denoising chain [Abstract].
The re-interpolation is compatible with the original flow-matching denoising process, so VINE preserves expressiveness and iterative generation without introducing a distinct sampler at inference [Abstract].
End-to-end backpropagation through all ten denoising steps is stable under VINE — a regime prior methods report unstable [Abstract].
VINE consistently outperforms state-of-the-art RL methods on the OGBench offline RL benchmark [Abstract].
VINE also outperforms state-of-the-art RL methods on a real-world robotic manipulation task [Abstract].

Method

VINE targets the specific failure mode of value-gradient RL for flow-matching policies. In a vanilla flow-matching policy trained with BC, the denoising chain integrates a learned velocity field starting from Gaussian noise; at inference, one Euler step per denoising timestep suffices because the loss enforces the velocity field is a smooth function of the interpolation state. When the training loss switches to a value gradient — i.e. the loss is -V(a_T) (or equivalent) evaluated at the final denoised action a_T, and the gradient is taken through the entire denoising chain — the same sampler becomes brittle: gradients propagate through a chain of Euler steps computed on a single sampled noise trajectory, and small perturbations at intermediate x_t compound multiplicatively into the value gradient at x_0 (the initial noise).

VINE’s fix is to replace the single-trajectory forward pass with a re-interpolation at every denoising step. Concretely, at each denoising step, rather than continuing along the trajectory produced by the previous step’s Euler update, VINE reconstructs a fresh interpolation state x_t = (1 − t)·x_0 + t·a_T_approx (or the closest equivalent under whatever flow-matching schedule is used), then computes the velocity from that reconstructed state. The key property is that this reconstructed state’s dependence on the policy parameters is direct — one linear interpolation — rather than the composed dependence of a full trajectory of Euler steps, so the value gradient traverses a stable path. Importantly, this modification is only made at training time: at inference, the sampler runs the standard flow-matching Euler procedure, preserving expressiveness and the multi-step generation regime.

Results

The paper’s abstract commits to three empirical claims (the body has the tables):

VINE stably backpropagates through all ten denoising steps of the flow-matching policy end-to-end and achieves stable policy improvement, in a regime where prior end-to-end methods report divergence [Abstract].
VINE consistently outperforms state-of-the-art RL methods on the OGBench offline RL benchmark [Abstract].
VINE outperforms state-of-the-art RL methods on a real-world robotic manipulation task; per-task success numbers are not in the abstract [Abstract].

Why it’s interesting

VINE takes the fourth possible position on the flow-matching-policy RL debate the wiki has been tracking. The existing entries on Reasoning RL and VLA Models triangulate the same problem — the flow-matching sampler is not designed to propagate a value gradient — with three distinct fixes: Test-Time Gradient Guidance of Flow Policies in Reinforcement Learning (QGF) refuses to update the policy at all and moves value guidance to inference; π*0.6: a VLA That Learns From Experience (RECAP) (RECAP/CFGRL) refuses to compute log-likelihoods and instead conditions supervised likelihood training on a binarized advantage token; Flow Matching Policy Gradients (FPO) keeps PPO but swaps the Gaussian likelihood ratio for a CFM-loss ratio, side-stepping BPTT. All three sacrifice something — inference-only means no policy update, CFGRL sacrifices per-step credit assignment, FPO sacrifices end-to-end backprop. VINE claims the sacrifice is unnecessary: fix the sampler and you can keep everything.

The diagnosis lines up interestingly with DiPOD: Diffusion Policy Optimization without Drifting Apart (DiPOD). DiPOD locates the instability at the objective-surrogate layer — the ELBO drifts from the true log-likelihood during policy improvement, and self-distillation keeps the two aligned. VINE locates it one layer further down, at the sampler layer — the trajectory-Euler forward pass gives value gradients an unstable path, and re-interpolation gives them a stable one. Both papers argue “the instability is not intrinsic to iterative diffusion policies; a specific piece of the pipeline is broken.” Neither paper cites the other; the natural experiment is to stack DiPOD’s ELBO-regularizer on top of VINE’s sampler and check whether the two fixes are redundant or additive — DiPOD attacks the update rule, VINE attacks the forward pass, so the priors point to additive.

The real-world robotic manipulation result matters for the VLA Models recipe board because it’s the first filed instance of end-to-end value-gradient RL beating the state of the art on a real robot without any of the three sacrifices. If the numbers hold in the body, VINE is a direct alternative to RECAP for the “how do you actually improve a flow-matching action head with RL” question that the concept page has been tracking — advantage-conditioning-as-supervision vs re-interpolated value-gradient BPTT is now a testable head-to-head.