Legato: Learning Native Continuation for Action Chunking Flow Policies

TL;DR

Legato is a project-page release proposing a training-time fix to the same async-inference / chunk-boundary problem that Real-Time Chunking (RTC) targets with test-time inpainting. Where RTC (and Train-Time RTC) treat the previously-executed action prefix as an inpaint condition at sampling time, Legato fine-tunes the flow-matching policy to natively continue from that prefix — the sharer’s framing is “same goal as RTC but not hacky at inference”. Across five real-world manipulation tasks (Bowls / Pour / PickPlace / Drawer / Towel) evaluated under matched pretraining, dataset, and hyperparameters, Legato reports lower completion time and smoother trajectories than RTC on every task, and beats Training-Time RTC on the Pour task specifically. Landing page only — no arxiv, code, or algorithmic detail beyond a comparative results table at filing time.

Key claims

Legato produces smoother trajectories than RTC across all five tasks, as measured by NLDLJ, NSPARC, and a chunk-overlap RMSE (roughly halved on Bowls, Pour, PickPlace, and Towel) [comparison table].
Legato produces shorter completion times than RTC on all five tasks (e.g. Pour 75.7 s vs 95.1 s, Towel 20.0 s vs 25.9 s), while matching or slightly exceeding task-score success [comparison table].
Legato beats Training-Time RTC on the Pour task specifically: score 9.72 vs 9.46, completion 75.7 s vs 81.7 s, NSPARC 1.65 vs 2.46 — the smoothness gap is larger than the success gap [training-time RTC table].
The training + evaluation protocol pins the same pretrained checkpoint, dataset, and hyperparameters across Legato and RTC, so the reported deltas are attributable to the continuation recipe rather than to pretraining or data [comparison-setup paragraph].

Method

Not disclosed on the project page. The framing (from the sharer’s note) is that Legato replaces RTC’s training-free inpainting-at-sampling with a fine-tuning objective that teaches the flow-matching policy to continue natively from an executed prefix. No architecture, loss, backbone, or hyperparameter details are provided at filing.

Results

Five real-world manipulation tasks, mean ± SE (matched pretrained checkpoint, dataset, hyperparameters):

Bowls: score 9.08 vs 8.68 (RTC); time 42.66 s vs 52.88 s; overlap RMSE 4.58 vs 6.83.
Pour: score 9.72 vs 9.34; time 75.73 s vs 95.07 s; overlap RMSE 5.14 vs 7.64; NSPARC 1.65 vs 2.85.
PickPlace: score 9.53 vs 9.47; time 30.37 s vs 35.53 s; overlap RMSE 5.98 vs 10.17.
Drawer: score 9.50 vs 9.20; time 21.80 s vs 25.97 s; NLDLJ 28.55 vs 32.73.
Towel: score 8.17 vs 7.33; time 20.00 s vs 25.93 s; overlap RMSE 6.22 vs 11.28.

Pour vs Training-Time RTC: 9.72/75.73/1.65/39.50 vs 9.46/81.73/2.46/39.95 (score / time / NSPARC / NLDLJ).

Why it’s interesting

Sits on the same async-inference / chunk-boundary axis as Real-Time Robot Execution with Masked Action Chunking (REMAC) and πR²: Reactive Real-time Flow Policies (πR²), all three of which name Train-Time RTC as the strongest prior baseline they compare against. The three occupy structurally distinct points: REMAC adds a LoRA + prefix-mask + self-conditioning training curriculum on top of a pretrained VLA and diagnoses intra-chunk inconsistency (executed prefix conditioned on a stale observation) as a separate failure from inter-chunk discontinuity; πR² rewrites the DiT’s AdaLN to per-position and installs a latency-adaptive staircase noise schedule so one denoising step per call emits new clean actions; Legato — per the sharer’s framing — fine-tunes the policy to natively continue the executed prefix, dispensing with RTC’s sampling-time inpainting altogether. The project-page metrics are the sharpest published smoothness-vs-RTC comparison at filing time (overlap RMSE and NSPARC both roughly halved on Pour and Towel), but the recipe itself is opaque without an accompanying paper. Also complementary to A2A: Action-to-Action Flow Matching (A2A: anchor one flow endpoint on the previous proprioceptive action for single-step Euler inference), which changes the starting distribution rather than the training objective — Legato appears to change what the policy is trained to output at chunk boundaries.