FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Control

TL;DR

FlashSAC is an off-policy actor-critic algorithm designed to displace PPO as the default RL recipe for sim-to-real high-dimensional robot control (humanoids, dexterous manipulation, vision-based). The paper argues that off-policy RL’s usual instability comes from bootstrapped critic updates compounding errors under scale, and resolves it with three mechanisms: (i) fast training via fewer larger updates on a large replay buffer + parallel simulation, (ii) stable training via explicit bounds on weight, feature, and gradient norms, and (iii) broad exploration via a unified entropy target and noise repetition. Reports beating PPO and FastTD3 across 60+ tasks in 10 simulators and cutting sim-to-real humanoid walking training from hours to minutes; won the Outstanding Paper Award at RSS 2026.

Key claims

Off-policy RL is the natural fit for high-dimensional control because it reuses replay data, but the bootstrapped Bellman objective is slow and unstable — targets depend on the critic’s own predictions so errors compound through repeated bootstrapping [§Introduction].
FlashSAC follows a supervised-learning-style scaling regime — massively parallel simulation, a large replay buffer, larger models with larger batches, and fewer gradient updates — which requires the stability mechanisms below to remain viable [§Fast training, Ablation].
Scaling alone worsens critic stability; FlashSAC bounds weight norms, feature norms, and gradient norms throughout training to keep bootstrapped updates stable at scale [§Stable training, Ablation].
The FlashSAC actor/critic architecture stacks inverted-residual blocks with pre-activation batch normalization and post-RMS normalization [§Architecture].
Exploration uses two complementary mechanisms — a unified entropy target and noise repetition — with ablations showing each is needed for broad state-action coverage [§Exploration, Ablation].
Evaluated across 60+ tasks in 10 simulators, FlashSAC outperforms PPO and strong off-policy baselines (including FastTD3) with minimal per-task tuning, and cuts sim-to-real humanoid walking from hours to minutes [§Results].

Method

FlashSAC starts from soft actor-critic and diagnoses three axes where naïve off-policy RL breaks at scale: throughput, stability, and exploration. The throughput axis is handled by scaling up in the supervised-learning direction — massively parallel simulators generate transitions into a large replay buffer, and the actor/critic are large models trained with large batches but fewer gradient steps per environment interaction, so the compute is spent on wide updates rather than frequent noisy ones. The stability axis, which normally breaks under that regime because the critic bootstraps against its own moving target, is handled by explicit bounds on the weight spectrum, on the intermediate features, and on the gradient — implemented through the inverted-residual actor/critic architecture with pre-activation batch normalization and post-RMS normalization. The exploration axis uses a unified entropy target plus noise repetition (holding the exploration noise over several steps rather than resampling every step), which broadens state-action coverage without additional tuning.

Results

Beats PPO and off-policy baselines (including FastTD3) across 60+ tasks spanning 10 simulators, with minimal per-task tuning [§Results].
Cuts sim-to-real humanoid walking training from hours to minutes vs prior recipes [§Results, project page].
Ablations attribute the wall-clock gains to the fast-training components (parallel sim, large replay + large model + large batches + fewer updates, code optimizations) and the stability gains to the norm-bound constraints [§Ablation studies].
Won the Outstanding Paper Award at RSS 2026 [author announcement].

Why it’s interesting

FlashSAC is the first filed paper to argue that the supervised-learning-style scaling recipe — big model, big batches, fewer updates, code optimization — transfers to online RL if you first bound the critic’s failure modes with weight/feature/gradient norm constraints. This is a stability recipe adjacent to the LLM training-stability lineage the wiki tracks under Xiaomi-Robotics-U0: Unified Embodied Synthesis with World Foundation Model‘s FP4 work and Composer 2 Technical Report‘s per-token FP32 activation scales, but applied to the actor-critic setting where the compounding error is the Bellman bootstrap rather than autoregressive precision loss. It also lands directly on the RL-for-robotics side of the concept page Reasoning RL, which so far has been dominated by GRPO-family LLM/VLM post-training — FlashSAC extends the “RL is fast, stable, and reusable” claim to sensorimotor policies where PPO has been the default. The most concrete near-term implication for filed work is on the sim-to-real bridge underneath the VLA Models concept: recipes like REGRIND: A Minimalist Retargeting-Guided RL Recipe for Dexterous Manipulation (REGRIND) and Flexion Reflect v1.0: The Path Towards Long-Horizon Autonomous Humanoid Work currently use PPO as the sim-side workhorse; FlashSAC’s claim is that a well-stabilized off-policy alternative dominates on both wall-clock and asymptotic performance.