WarpSAC: Towards the Pinnacle of Scalable Off-Policy RL by Rethinking Exploration and Exploitation

TL;DR

WarpSAC is a regime-aware extension of FlashSAC arguing that the stabilizers off-policy RL uses to keep bootstrapped Bellman updates in check are not universal — their sign flips with data availability. Controlled experiments across eight benchmark families show that parameter-projection normalization helps under narrow replay coverage (CPU-scale training) but restricts value fitting when data are abundant (GPU-parallel training), and that clipped double-Q can be safely relaxed to single-Q under high throughput. WarpSAC packages these into two profiles sharing a common Sample Weight Decay (age-biased replay weighting) exploitation mechanism: WarpSAC-L (Norm ON, clipped double-Q) for CPU-scale and WarpSAC-A (Norm OFF, single-Q) for GPU-parallel. Reports 4.5%/23.1% AUC gains over FlashSAC on nine CPU-scale and fourteen GPU-parallel environments respectively, and lifts UnitreeG1TransportBox success from 19.8% to 96.4%.

Key claims

Off-policy RL stabilizers are data-regime-dependent, not universal: parameter normalization helps under narrow replay coverage (CPU-scale) but restricts value fitting when data are abundant (GPU-parallel), and clipped double-Q can be relaxed to single-Q in high-throughput manipulation [§Abstract, §Controlled experiments].
Age-biased replay weighting (Sample Weight Decay, SWD) improves learning efficiency across regimes, especially with limited network capacity — sampling recent policy-relevant transitions more often while retaining older transitions with nonzero weight for coverage; SWD introduces no auxiliary network, Bellman target, or loss term, only changing the minibatch distribution [Project page §Method].
WarpSAC provides two profiles sharing the SWD replay mechanism but distinguishing the network-side stabilizer: WarpSAC-L (Norm ON, clipped double-Q) for data-limited CPU-scale training, WarpSAC-A (Norm OFF, single-Q) for data-abundant GPU-parallel training; profile defaults resolve from environment type and workload scale [Project page §Method].
Parameter-projection normalization renormalizes network parameters after each optimizer step, constraining the effective function class and reducing unstable value extrapolation — but the effect is regime-dependent: the same constraint that stabilizes narrow-coverage replay restricts value fitting when the replay is broad [Project page §Method].
Reports +4.5% normalized-score-step AUC over FlashSAC across nine CPU-scale environments and +23.1% across fourteen GPU-parallel environments, +19.1% mean normalized wall-time AUC on MuJoCo Playground, and lifts UnitreeG1TransportBox-v1 success from 19.8% to 96.4% [§Abstract].
Reports 36.4% faster sim-to-real deployment on Unitree G1 humanoid than FlashSAC [§Abstract].

Method

WarpSAC keeps FlashSAC’s SL-style scaling recipe (parallel simulation + large replay + large model + large batches + fewer updates) but re-examines two components empirically. First, on the replay side, Sample Weight Decay (SWD) assigns each transition an age-dependent sampling weight so recent policy-relevant data are oversampled while older transitions retain nonzero weight — this changes only the minibatch distribution, adds no auxiliary head or loss, and is applied identically in both profiles. Second, on the network side, parameter-projection normalization (renormalization of network parameters after each optimizer step) and clipped double-Q are treated as profile-level choices rather than universals. WarpSAC-L (CPU-scale profile) enables both: normalization ON + clipped double-Q, matching the FlashSAC prior. WarpSAC-A (GPU-parallel profile) disables both: normalization OFF + single-Q, on the argument that when replay coverage is broad the value function must be free to extrapolate and the pessimism from double-Q clipping becomes an unnecessary tax. Automatic profile selection resolves defaults from the environment type and workload scale.

Results

+4.5% normalized-score-step AUC over FlashSAC across nine CPU-scale environments [§Abstract].
+23.1% normalized-score-step AUC over FlashSAC across fourteen GPU-parallel environments [§Abstract].
UnitreeG1TransportBox-v1 success rate: 19.8% → 96.4% [§Abstract].
+19.1% mean normalized wall-time AUC on MuJoCo Playground [§Abstract].
36.4% faster sim-to-real deployment on Unitree G1 vs FlashSAC [§Abstract].

Why it’s interesting

WarpSAC is the direct successor to FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Control (RSS 2026 Outstanding Paper) and refines FlashSAC’s most confident recipe knob — “always bound weight/feature/gradient norms to keep bootstrapped critic updates stable at scale” — into a conditional one: the norm-bound stabilizer that made FlashSAC work at CPU scale is exactly what restricts value fitting at GPU-parallel scale, and the same holds for clipped double-Q. This is a data-regime version of the training-stability-recipe conversation on Reasoning RL currently dominated by LLM-precision debates (Defeating the Training-Inference Mismatch via FP16, The 4-bitter Lesson: Balancing Stability and Performance in NVFP4 RL) — the “correct” stabilizer depends on how much data the training loop sees per gradient step, not on any universal rule. On the VLA Models side, WarpSAC’s Unitree G1 sim-to-real numbers (36.4% faster deployment, 96.4% TransportBox success) are a stronger sensorimotor RL datapoint than the LLM/VLM GRPO recipes that dominate the concept page, and reinforce the FlashSAC bet that a well-stabilized off-policy alternative dominates PPO on the sim-to-real bridge that recipes like REGRIND: A Minimalist Retargeting-Guided RL Recipe for Dexterous Manipulation and Flexion Reflect v1.0: The Path Towards Long-Horizon Autonomous Humanoid Work use.