A2A: Action-to-Action Flow Matching

ListenDon't Start From Noise: Action-to-Action Flow Matching10:24

Show notes

This week we dig into **A2A: Action-to-Action Flow Matching** (RSS 2026), which throws out the Gaussian-noise starting point of diffusion and flow-matching robot policies and instead initializes the flow from the robot's *previous* action. Because adjacent action chunks are physically continuous, the flow trajectory is short — a single Euler step gets to 0.56 ms latency on an RTX 5090 while holding ninety percent–plus success. We talk through what the numbers actually show, the visual-perturbation and out-of-distribution results, and why the "adjacent latents are close" idea might carry over to video generation. **Sources** - A2A: Action-to-Action Flow Matching

Subscribe to the channel: https://podcast.lumalabs.link/27a25f51ea81d41a8253f46c6133a35e/feed.xml

TL;DR

A2A (Action-to-Action Flow Matching) replaces the Gaussian-noise starting point of diffusion / flow-matching robot policies with an initialization drawn from the previous proprioceptive action, embedded into the same high-dimensional latent as the target action chunk. Because adjacent action chunks are physically continuous, the flow trajectory between them is short, and a single Euler step suffices — the paper reports 0.56 ms inference latency on RTX 5090 in the single-step regime with ≥90% task success, beating multi-step Diffusion Policy / flow-matching baselines on five simulation manipulation tasks and outperforming them under heavy visual randomization and unseen configurations. Presented as an RSS 2026 paper; also extends the same “start from a temporally adjacent latent” idea to video generation as a broader temporal-modeling principle.

Key claims

Traditional diffusion policies inherit the “start from uninformed noise” assumption from image synthesis, but this is a structural mismatch for robots: proprioceptive feedback is continuous, adjacent action chunks are physically similar, and history can serve as a strong initialization for the flow [§ “Key Insight”].
A2A converges to 100% success within 40 training epochs and reaches 92% / 92% / 86% / 92% / 90% on Close Box / Pick Cube / Stack Cube / Open Drawer / Pick-Place Bowl at 6 steps and 100 demos, vs 82–58% / 88–58% / 80–16% / 90–14% / 92–68% for VITA / FM-UNet / FM-DiT / DDPM-UNet / DDPM-DiT / DDIM-UNet / Score-UNet baselines [Quantitative Comparison table].
Success saturates at 4 inference steps and remains ≥90% at a single step after 32 epochs; mean latency ~1 ms and 0.56 ms in single-step on RTX 5090 [§ “Ingredient 2”].
Under progressive visual perturbations (new backgrounds → lighting → viewpoints), A2A degrades from 92% to 38% at 6 steps and 90% to 32% at 1 step, vs FM-UNet 70% → 16% (10 steps) and DDPM-UNet 60% → 10% (100 steps) — the largest surviving margins in the visual-generalization comparison [Visual Generalization Results table].
On an out-of-distribution real-world “glowing cube” variant of Pick Cube, A2A holds 80% success while FM-UNet and DDPM-UNet drop to 0% [§ “Ingredient 3”, (b)].
Injecting small Gaussian noise into the historical action initialization is presented as a mechanism for generalizing to unseen starting configurations without retraining [§ “Initial State Generalization”].
The same action-to-action starting-point idea is claimed to extend to video generation as a general temporal-modeling technique [§ Abstract].

Method

A2A reformulates the flow-matching policy so that the noise/prior end of the ODE trajectory is not $\mathcal{N}(0, I)$ but a learned embedding of the recent proprioceptive action history. Concretely, past action sequences are lifted into the same high-dimensional latent space as the target action chunk, and the flow-matching velocity field is trained to transport this history-conditioned prior to the next action chunk. Because the two endpoints are close in latent space (adjacent chunks differ by only one control interval), the Euler discretization of the ODE is well-behaved at very small NFE — the paper’s headline claim is that a single Euler step suffices for competitive success rates. At deployment the robot’s live proprioceptive stream produces the initialization on every step, and small Gaussian perturbations on that initialization are used to widen the effective start distribution at test time for OOD configurations. The blog page reports evaluations across five Robosuite tasks in simulation and matched real-world validation with 30 demonstrations, and re-uses the same “temporally adjacent latent as start” idea in a video-generation ablation whose details are deferred to the paper.

Results

Sim (100 demos, 30 epochs, 5 tasks): A2A at 6 NFE is best or tied on every task (92/92/86/92/90), beating VITA at 6 NFE (88/88/80/90/92) with the largest margins on Stack Cube (+6) and Close Box (+4). Multi-step DDPM baselines lag by 20–70 points on Stack Cube and Open Drawer at 40–100 NFE [Quantitative Comparison table].
Single-step regime: A2A reaches ≥90% success at 1 NFE after 32 epochs — matches ACT (1 NFE, 82/86/32/80/60) and outperforms 100-NFE DDPM baselines [§ “Ingredient 2”, Quantitative Comparison table].
Latency: ~1 ms mean inference at 4–6 NFE; 0.56 ms at 1 NFE on RTX 5090 [§ “Ingredient 2”].
Visual generalization (30-demo regime, Pick Cube): at Level 3 perturbations (new backgrounds + lighting + viewpoints), A2A holds 38% at 6 NFE and 32% at 1 NFE, vs FM-UNet 16% and DDPM-UNet 10% at their native step counts [Visual Generalization Results table].
Real-world OOD: on a glowing-cube variant of Pick Cube, A2A holds 80% success while FM-UNet and DDPM-UNet collapse to 0% [§ “Ingredient 3”, (b)].
Baseline set: VITA, FM-UNet, FM-DiT, DDPM-UNet, DDPM-DiT, DDIM-UNet, Score-UNet, ACT (each at its native step count). Table is per-task success across 50 rollouts per cell.

Why it’s interesting

A2A is a from-scratch counter-recipe to the wiki’s dominant distillation-based answer for fast flow-matching action heads: rather than compress a many-step teacher (as in Transition Matching Distillation for Fast Video Generation TMD or TDM-R1: Reinforcing Few-Step Diffusion Models with Non-Differentiable Reward TDM-R1), it shortens the flow trajectory itself by anchoring one endpoint to proprioception. If the reported 0.56 ms / 1 NFE numbers hold, it competes on the same latency axis as the B-spline Policy B-spline Policy: Accelerating Manipulation Policies via B-spline Action Representations which achieves 2–4× speedups by changing the action output space — A2A instead changes the starting distribution of the flow, and the two axes should compose. The recipe also sits orthogonal to the “frozen action head + tiny steering adapter” thread (FlowDAgger: Human-in-the-Loop Adaptation of Generative Robot Policies in Latent Space FlowDAgger) and to the causal-forcing / DiagDistill line for AR video (Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generation, Streaming Autoregressive Video Generation via Diagonal Distillation) which similarly exploit temporal continuity to shorten trajectories — a shared “adjacent latents are close, so the flow between them is short” principle now surfacing across VLAs and video.