Residual Modeling Closes the Regression and Generative Policy Gap in Robot Learning

TL;DR

HT-Policy argues the regression-vs-generative-policy gap in behavior cloning is not fundamentally about action multimodality but about a mismatch between MSE’s fixed-scale Gaussian assumption and the heavy-tailed, state-dependent-scale residuals real demonstrations actually produce — under MSE, high-residual observations capture disproportionate gradient magnitude, hurting optimization. Replacing MSE with a heteroscedastic Student-t loss (state-dependent scale + Student-t noise model) makes a single-pass feed-forward regression policy competitive with flow-matching baselines across LIBERO, RoboCasa-GR1, and LIBERO-10, while running 1.82–17.83× faster at inference. The method is drop-in for existing flow-matching backbones (GR00T N1.7, π0.5, Cosmos3-Nano) and reuses pretrained weights, offering an efficient direct-regression alternative on the same architectures.

Key claims

Action-prediction residuals in real demonstration data show substantial state-dependent variation in scale and heavier-than-Gaussian tails, violating MSE’s fixed-scale Gaussian assumption [§Analysis of real-world data].
Both MSE and flow-matching policies exhibit heavy-tailed action residuals, but their training gradients behave differently: MSE allocates disproportionate gradient magnitude to high-residual observations, hurting optimization — a plausible explanation for why denoising objectives help by predicting injected Gaussian noise instead of raw actions [§Gradient analysis].
HT-Policy — heteroscedastic Student-t regression with input-dependent scale heads — predicts action chunks in a single feed-forward pass and matches flow-matching baselines on four simulation benchmarks and real-robot evaluations [§Experiments].
HT-Policy can reuse pretrained flow-matching backbones (GR00T N1.7, π0.5, Cosmos3-Nano) rather than requiring new policy architectures [§Method].
Inference efficiency vs flow-matching baselines: GR00T N1.7 1.88× faster (40.78 FPS vs 21.65), π0.5 1.82× faster (14.72 vs 8.09), Cosmos3-Nano 17.83× faster (14.02 vs 0.79) [Twitter thread, HT-Policy vs Flow table].
Task-success parity with flow matching at matched compute: GR00T N1.7 on RoboCasa-GR1 50.0% HT vs 42.3% Flow; π0.5 on LIBERO 97.0% vs 97.1%; Cosmos3-Nano on LIBERO-10 96.5% vs 95.6% [Twitter thread, Performance table].

Method

The paper reframes the behavior-cloning action-head design choice as a statistical model mismatch problem rather than a multimodality problem. Standard MSE regression implicitly assumes action residuals follow a fixed-scale Gaussian — but empirical analysis of robot demonstration data shows two violations: (1) residual scale varies systematically across task stages (pushing has higher flexibility than picking, so residuals are larger), and (2) the residual distribution is heavy-tailed rather than Gaussian. Under this mismatch, MSE training gradients concentrate on the high-residual tails, which the authors argue is why MSE-trained regression policies optimize poorly.

HT-Policy replaces the MSE loss with a heteroscedastic Student-t negative log-likelihood: the network predicts both the action chunk and a per-observation scale (via a learned scale head), and the Student-t noise model gives finite influence to heavy-tail observations. Both components are ablated as essential — removing either collapses performance. The architecture reuses flow-matching backbones directly (so pretrained GR00T N1.7, π0.5, and Cosmos3-Nano weights transfer), but swaps the flow-matching denoising head for a single-pass regression head with the HT loss on top.

The analysis-side contribution is a gradient decomposition showing that flow-matching’s denoising objective reduces heavy-tail influence by predicting injected Gaussian noise (which is by construction well-behaved) rather than raw actions — providing a mechanism for why flow matching outperforms MSE that is orthogonal to action-space multimodality.

Results

Across four simulation benchmarks and real-robot evaluations, HT-Policy matches or edges out flow-matching baselines at matched compute while running 1.8–17.8× faster:

GR00T N1.7 / RoboCasa-GR1: 50.0% HT vs 42.3% Flow (+7.7 pp).
π0.5 / LIBERO: 97.0% vs 97.1% (parity).
Cosmos3-Nano / LIBERO-10: 96.5% vs 95.6% (+0.9 pp).

Inference speedups (reported as FPS on unspecified hardware in the author’s Twitter thread):

GR00T N1.7: 40.78 FPS HT vs 21.65 Flow — 1.88×.
π0.5: 14.72 FPS vs 8.09 — 1.82×.
Cosmos3-Nano: 14.02 FPS vs 0.79 — 17.83×.

The Cosmos3-Nano gap is the sharpest datapoint: the flow-matching baseline on this backbone runs at under 1 FPS, which HT lifts to ~14 FPS without quality loss, suggesting the biggest wins are on backbones where flow matching is already bottlenecked by multi-step denoising. Both the state-dependent scale head and the Student-t noise model are ablated as load-bearing — neither alone recovers full performance.

Why it’s interesting

HT-Policy sits on the same action-head design axis as TemporalFlow-VLA: Learning Physically Grounded Execution History for Long-Horizon Robot Manipulation TemporalFlow-VLA, Action QFormer: Structured Representation Shaping under Action Supervision in Vision-Language-Action Models Action QFormer, and Modality-Autoregressive World-Action Models ModAR but attacks it from a different angle: rather than restructuring the architecture, interface, or training recipe, it swaps the loss function and keeps the flow-matching backbone intact. The gradient-decomposition analysis gives a mechanistic alternative to the folk “flow matches multimodal action distributions” story on VLA Models, and the result sharpens the recipe-lever board — the generative-vs-regression question may collapse into a loss-engineering question for a large class of tasks.

Direct companion to the Behavioral Cloning Mystery “Behavioral Cloning Mystery” line and Why Does Action Chunking Improve Behavioral Cloning Performance in Robotic Control? action-chunking analysis: both diagnose counterintuitive BC phenomena at the data+objective level without perception as a confounder, and HT-Policy adds a specific loss-side fix for one of them (heavy-tailed residuals breaking MSE optimization). The 17.83× speedup on Cosmos3-Nano also contrasts cleanly with the few-step distillation line (Few-Step Generation via Data-Space Iteration, ImageWAM: Do World Action Models Really Need Video Generation, or Just Image Editing?, C³ache: Accelerating World Action Models with Cross Inference Chunk Cache) — pursuing single-pass regression rather than reducing denoising steps.