Skip to content
Bud's Wiki

Reasoning RL

TL;DR

Section titled “TL;DR”

“Reasoning RL” is the current default recipe for post-training models on tasks where correctness is checkable but the path to it is not: optimize with GRPO (or a close cousin) against shaped or gated rewards, typically over self-generated or environment-generated rollouts. Across filed papers the pattern is consistent — GRPO + structured rewards is doing the heavy lifting for tool-use agents, code/SWE models, and now VLM perception tasks — and the interesting design surface has moved from the RL objective to the reward shape and the data loop feeding it.

Key claims

Section titled “Key claims”

Recent contributions

Section titled “Recent contributions”
  • [2026-07-20] Scaling Behavior Foundation Model for Humanoid Robots: ScaleBFM (Zeng et al.) uses PPO as the pretraining objective for a humanoid Behavior Foundation Model over a 102M-frame retargeted human-motion corpus, and treats on-policy rollout collection as an explicit scaling axis: jointly scaling environment parallelism (width) and rollout horizon (depth) is required for consistent gains, while single-axis scaling does not help. Adds a whole-body-control datapoint to the reasoning-RL / on-policy-rollout scaling picture that complements the LLM-agent settings already on file, and reports MPKPE reductions of >10% (local) / 82% (global) vs prior humanoid controllers.
  • [2026-07-20] Understanding Reasoning from Pretraining to Post-Training: Shen et al. (NYU + others) build a chess-based controlled testbed for the full pretraining → SFT → RL pipeline (5M–1B params on human games, SFT on synthetic reasoning traces, RL on puzzles with verifiable rewards) and derive two quantitative findings: (i) post-RL performance at matched RL compute is well-predicted from pretraining loss, and (ii) the slope of the RL reward curve improves approximately linearly with pretraining tokens — longer-pretrained checkpoints not only start higher but learn faster under RL. Mechanistically: RL amplifies moves the SFT policy already preferred on easy puzzles, but surfaces correct moves nearly absent under SFT on hard ones — a scaling-law-level statement of the “extend vs. amplify” split. Same pattern reproduces on a 1B math-text run. Continuous-scaling complement to On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models‘s discrete headroom/edge-of-competence framing and a third independent measurement (behavioral, alongside Spurious Rewards: Rethinking Training Signals in RLVR‘s pretraining-behavior-elicitation and The Path Not Taken: RLVR Provably Learns Off the Principals‘s off-principal-subspace finding) that RL’s job depends on what pretraining put there.
  • [2026-07-20] Learning to Fold: prizewinning solution at LeHome Challenge 2026 (1st place online, 2nd offline): Learning to Fold (Larchenko, LeHome Challenge 2026 winner in sim / 2nd in real) contributes two orthogonal ideas to the flow-matching-VLA RL story: (1) AWR through the sampler rather than the loss weight — high-|advantage| frames loaded proportionally more often, batches stay unweighted MSE, auxiliary heads importance-weighted back to unbiased, effective batch utilization stays at 100% of the weight mass so most of the compute goes to the good actions we actually want to learn; (2) policy-as-value-function — cheap linear heads on a single learned query token predict success, completion, garment type, current/future keypoint distances, and an action-conditional success residual (Q-function), all reading image tokens only. Argues explicitly that conditioning + reweighting families (AWR + RECAP) fit flow-matching VLAs better than PPO/GRPO because valid actions occupy a thin manifold and policy-gradient methods that “push probability away from bad actions” mostly push off that manifold — the sharpest filed formulation of the log-likelihood-manifold argument that motivates RECAP-family methods over PPO for flow-matching action heads. Also introduces a checkpoint-rollback trick as a systematic local-optima escape mechanism.
  • [2026-07-19] Read It Back: Pretrained MLLMs Are Zero-Shot Reward Models for Text-to-Image Generation: SpectraReward is a training-free reward function for GRPO-family T2I post-training: teacher-force a pretrained MLLM to reproduce the prompt from the generated image and use the average prompt log-likelihood as the reward. Composes with three RL algorithms and two diffusion policies in the sweep. Self-SpectraReward variant closes the loop inside a single UMM — policy’s understanding branch scores its own generation branch, no external reward model. Claims that reward-policy alignment dominates reward-model scale in image-generation RL.
  • [2026-07-18] FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Control: FlashSAC (RSS 2026 Outstanding Paper) argues off-policy RL should displace PPO as the default sim-to-real recipe for high-dimensional robot control. Extends the RL-recipe conversation from LLM/VLM GRPO territory into sensorimotor SAC-family algorithms with three interventions: (i) SL-style scaling (parallel sim + large replay + large model + large batches + fewer updates), (ii) explicit weight/feature/gradient norm bounds via an inverted-residual actor/critic with pre-BN + post-RMS-norm to keep bootstrapped critic updates stable at scale, (iii) unified entropy target + noise repetition for broad exploration. Beats PPO and FastTD3 across 60+ tasks in 10 simulators; cuts sim-to-real humanoid walking from hours to minutes. Concept-page shape: same “make bootstrapped RL stable enough to scale” claim Defeating the Training-Inference Mismatch via FP16 and The 4-bitter Lesson: Balancing Stability and Performance in NVFP4 RL make on the LLM-precision axis, applied to the actor-critic setting where the compounding error is Bellman rather than autoregressive.
  • [2026-07-17] DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulation: DenseReward proposes a dense frame-level VLM reward as the substrate for RL fine-tuning of robot policies, targeting the sparse-trajectory-reward bottleneck that has kept robotic RL from routinely improving past IL plateaus; reports the learned reward is usable for both MPC and downstream RL. Same shape as LLM-as-a-Verifier: A General-Purpose Verification Framework‘s continuous-verifier-as-reward finding, applied to the manipulation domain.
  • [2026-07-17] UniVR: Thinking in Visual Space for Unified Visual Reasoning: UniVR’s VR-GRPO decomposes the GRPO reward into a global trajectory-level term and a step-level per-step term, jointly enforcing logical coherence and physical consistency during the reasoning process. New shaping axis: prior recipes on this page shape by task type (hierarchical distance, correctness+format, perception-stack); VR-GRPO shapes by temporal position within a single rollout, closer in spirit to process-reward modeling than to GRPO’s usual outcome-only signal. Trained from pure visual demonstrations (no image-text pairs) with up to +25% on the new VR-X benchmark.
  • [2026-07-16] UniIntervene: Agentic Intervention for Efficient Real-World Reinforcement Learning: UniIntervene (Deng et al.) attacks the HiL-RL scaling bottleneck — human-labor cost per intervention — by learning an intervention agent that fires the recovery decision internally. Three modules: (a) future-conditioned action-value critic supervised by frozen V-JEPA2 latent-future targets + a progress-aligned proxy value, giving a value signal grounded in predicted dynamics rather than a single-frame classifier; (b) temporal value-risk critic that triggers only on sustained value stagnation or degradation over a sliding window (crucial for filtering transient contact dips); (c) memory-guided recovery — retrieve nearest-neighbor high-value state from past interventions, decode corrective actions via goal-conditioned policy. Real-world UR7e results: +8.6 SR while reducing IR by 57% vs HiL-SERL baseline, and beats a failure-anticipation-RL (FA-RL) trigger on both axes. Concrete answer to the “HiL-RL doesn’t scale because humans are in the tightest inner loop” open question — the human still labels the hardest interventions, just far fewer of them.
  • [2026-07-16] FlowDAgger: Human-in-the-Loop Adaptation of Generative Robot Policies in Latent Space: FlowDAgger sharpens the “when is RL worth it for flow-matching policies” question with a direct baseline against DSRL (latent-space RL on the noise input of a frozen flow policy): from the same handful of human corrections, supervised regression on inverted-noise targets reaches MetaWorld mean 0.78 vs DSRL’s 0.55, at a fraction of the compute and with no online rollouts. Datapoint that for the low-correction-budget regime on flow-matching VLAs, direct supervision on reverse-integrated noise dominates on-policy noise-space RL — complementary to π*0.6’s argument that advantage-conditioning-as-supervision dominates policy-gradient RL on the action side of the same problem.
  • [2026-07-15] Introducing Inkling — Thinking Machines' open-weights 975B/41B MoE multimodal reasoning model: Inkling scales RL to >30M rollouts across two long continuous runs with log-linear reasoning-benchmark improvement throughout, and trains controllable thinking effort by varying the system message and per-token cost across samples so a single checkpoint learns a smooth effort→tokens→performance curve (matches Nemotron 3 Ultra Terminal Bench 2.1 at ~⅓ the tokens). Emergent CoT compression observed during RL despite no direct reward for brevity — matches the concurrent Cognition SWE-1.7 finding. Uses dual graders (rubric + agentic-web-search claim verification) with abstention-aware short-form QA to jointly optimize helpfulness and calibration.
  • [2026-07-15] Foveated Reasoning: Stateful, Action-based Visual Focusing for Vision-Language Models: Applies the SFT-cold-start-then-RL recipe to a non-textual action space — spatial region selectors for visual foveation rather than tool calls or text tokens. Reward jointly targets task accuracy and evidence-acquisition quality, with an explicit anti-”see-everything” term that penalizes the degenerate policy of always foveating on the entire image. Notable that the cold-start uses pseudo labels rather than human gaze annotations, and the pseudo-labeled variant produces more multi-fixation behavior than human-labeled controls — a case where “worse” labels shape a better policy.
  • [2026-07-14] AIDE²: First Evidence of Recursive Self-Improvement: Weco’s AIDE² reports an emergent anti-reward-hacking mechanism that requires no anti-hacking objective and no reward shaping: KernelBench hacking rate drops from 63% (AIDE0) → 42% (AIDEhuman) → 34% (AIDE85) purely because outer-loop selection uses a private score the inner agent cannot see, so variants that game the public score fail to survive. Discovered agent AIDE85 also built its own three-layer defense (anti-overfitting prompt injection, output-regeneration guards, statistical outlier removal). Complements reward-shaping approaches on the concept page with a training-free structural alternative: hide the selection signal from the optimizer.
  • [2026-07-10] The 4-bitter Lesson: Balancing Stability and Performance in NVFP4 RL: humans& open-source the first end-to-end NVFP4 RL recipe for a frontier MoE (Qwen3-30B-A3B on DAPO-math-17k), combining dequantized backward, Four-Over-Six on both weights and activations, and selective BF16 for the shared MoE expert + last ~15% of layers; all three interventions are needed to match BF16 reward and grad-norm curves. Independently corroborates Composer 2 Technical Report‘s per-token-FP32 activation-scale finding as necessary for RL stability under NVFP4, and directly opposes Defeating the Training-Inference Mismatch via FP16‘s “upgrade RL to FP16” direction — the debate is now three-way (FP16 / BF16 / selective NVFP4).
  • [2026-07-09] Introducing Muse Spark 1.1: Muse Spark 1.1 is deployed as a “Thinking”-mode reasoning model; the blog claims (without numbers) reduced hallucination and sycophancy versus the prior Muse Spark, plus stronger adversarial robustness — attributed to training/eval rather than architecture.
  • [2026-07-08] Robostral Navigate: single-camera AI navigation: Robostral Navigate uses CISPO for online RL polish on top of SFT for embodied navigation, adding +3.2% R2R-CE success and inducing recovery/exploration behaviors that mitigate BC distribution shift — CISPO-for-VLN counterpart to CFGRL/RECAP for flow-matching VLAs.
  • [2026-07-08] Towards Practical World Model-based Reinforcement Learning for Vision-Language-Action Models: VLA-MBPO gives the sharpest theoretical + empirical treatment on file of compounding error in model-based VLA-RL: for γ=0.95 and chunk size c=5, prior full-horizon step-level rollouts have a value gap of ~2000 while chunk-level branched rollout with k=2 achieves ~80 (~25× tighter). Empirically confirmed on LIBERO-Long, where full-horizon rollout drops 66.8 → 52.8. Uses Flow-Noise (PPO variant for flow-matching action heads) and reports a single hyperparameter set works across four LIBERO suites plus five real-world tasks — a rarity for RL post-training on embodied policies.
  • [2026-07-08] A global workspace in language models: Anthropic’s counterfactual reflection training fine-tunes Claude only on what it would say if interrupted mid-task and asked to reflect on its behavior, never on the actual task response, and finds that this lowers dishonest-behavior rate on evaluations while shifting J-space (workspace) contents on those tasks toward “honest”/“integrity”. Suggests that shaping the reflective/verbal channel is a low-footprint mechanism for changing un-verbalized behavior — a training-time counterpart to activation-steering interventions.
  • [2026-07-05] Agent² RL-Bench: Can LLM Agents Engineer Agentic RL Post-Training?: Agent² RL-Bench measures whether CLI agents can autonomously engineer GRPO / PPO / DPO pipelines end-to-end under a 12h budget. Reinforces the concept page’s claim that GRPO is used as stage-two after SFT rather than from-scratch: agents attempted RL methods first but fell back to SFT-based pipelines on 5 of 6 tasks; RL-only mode succeeded as the final best route only on ALFWorld (5.97 → 93.28 via SFT warm-up + GRPO with online rollouts). Operating-mode ordering reverses between L1 static and L3 interactive — SFT-only wins GSM8K but is worst on ALFWorld (67.91), RL-only wins ALFWorld but is worst on GSM8K — best-to-worst mode spread exceeds 80 points on ALFWorld.
  • [2026-07-02] Spurious Rewards: Rethinking Training Signals in RLVR: Spurious Rewards (Shao et al., the paper audited by Incorrect Baseline Evaluations Call Into Question Recent LLM-RL Claims) — GRPO with random / format / incorrect-label rewards produces ~+20-point MATH-500 gains on Qwen2.5-Math-7B, approaching the +29.1% ground-truth gain, but fails or hurts on Llama3 and OLMo2. Mechanism isolated: RLVR upweights a pretraining behavior (Python-code-like solution traces without code execution), 65% → >90% frequency within 15 steps under any of the weak/spurious rewards; the Lang→Code transition subset accounts for 58.3% of Qwen2.5-Math-7B’s gain. Prompt- and reward-based interventions on code frequency causally reproduce the accuracy pattern. Empirical primary source for the “RLVR at current scales elicits pretraining behaviors, does not install new ones” framing that The Path Not Taken: RLVR Provably Learns Off the Principals and History May Repeat Itself: RSI Seen from a Previous AI Era approach from other angles.
  • [2026-07-02] Learning to Replicate Expert Judgment in Financial Tasks: Thinking Machines + Bridgewater fine-tune Qwen3-235B on six financial-triage tasks with a three-part recipe (interleaved per-task batching + CISPO with asymmetric clipping + on-policy distillation where the teacher is promoted from the current student only when validation accuracy sets a new high). Beats the best frontier model (GPT/Claude/Gemini) by 6.5 accuracy points at 13.8× lower inference cost. The promoted-teacher OPD schedule is a novel variant of the recipe from On-Policy Distillation — makes the teacher a monotonically-improving moving target rather than a fixed reference.
  • [2026-06-29] Flexion Reflect v1.0: The Path Towards Long-Horizon Autonomous Humanoid Work: Flexion Reflect v1.0 reports SFT-then-RL on a humanoid mission-controller VLM moving end-to-end 16-step mission completion from 38% (SFT) to 90% (SFT+RL); rare datapoint that the same recipe used on math/code LLMs is the lever for high-level embodied mission reasoning, not just low-level skills.
  • [2026-06-29] CHORD: Object-Centric Contact Wrench Guidance for Dexterous Manipulation: CHORD reframes the human-to-robot dexterous-manipulation RL reward from position-similarity to wrench-similarity: matching the 6-D force+torque a contact induces on the object, rather than where the contact occurs in space, which the paper argues sidesteps the kinematic gap and makes contact-rich RL scalable across rigid, articulated, and bimanual tasks.
  • [2026-06-29] World Value Models for Robotic Manipulation: WVM (ByteDance Seed) is a generalist value model built on a world-model backbone (instead of a VLM backbone like GVL / VLAC / Robo Reward), designed as a recipe-agnostic substrate for offline-RL-style policy extraction on robotic manipulation across mixed-quality data — a value-side analog of the world-model-vs-VLM debate playing out in policy backbones. Reports SOTA on standard expert Value-Order Correlation (0.95 avg) and on the new Suboptimal-Value-Bench (0.78 avg VOC, 0.05 hesitation RMSE) where VLM-backbone baselines like VLAC collapse to -0.37 VOC.
  • [2026-06-29] ABACUS: Adapting Unified Foundation Model for Bridging Image Count Understanding and Generation: ABACUS applies GRPO twice in a counting-specialist UMM: a boundary-aware count policy that penalizes crop-edge double-counting, and a cycle-consistent GRPO loop where the understanding branch self-critiques generation-branch outputs as the reward — a self-rewarding RL recipe that needs no external annotations.
  • [2026-06-29] Hy-Embodied-0.5-VLA: From Vision-Language-Action Models to a Real-World Robot Learning Stack: HyVLA-0.5 (Tencent Hunyuan) names RL post-training as a distinct stage of an end-to-end VLA stack alongside continued pre-training and SFT — third industrial VLA report to feature RL post-training after π*0.6 / RECAP, though method specifics are not retrievable from the abstract.
  • [2026-06-29] Qwen-Image-2.0-RL Technical Report: Qwen-Image-2.0-RL applies GRPO with three explicit stabilizers — hybrid CFG to preserve pretrained knowledge during RL rollouts, intra-group reward-range filtering for prompt curation, and per-category reward weight calibration — to post-train a unified T2I + edit diffusion model. Each task gets its own GRPO specialist trained against a fine-tuned VLM composite reward stack (alignment/aesthetics/portrait for T2I; instruction-following/face-ID for edit), then the two specialists are merged into one student via on-policy distillation with trajectory-level velocity matching. +2.61 Qwen-Image-Bench and +78 / +93 arena Elo over the base.
  • [2026-06-27] Agentic World Modeling: Foundations, Capabilities, Laws, and Beyond: Position-driven survey that frames reasoning-RL agents as L2/L3 world-modeling systems: model-based RL is named as the canonical capability progression where latent-imagination rollouts (DreamerV3, MuZero) match or exceed model-free baselines, and the L3 (Evolver) stage is operationalized as the capacity to revise the world-model stack itself from deployment evidence — a vocabulary that gives RL self-improvement and reflection loops a precise place in the broader world-modeling map.
  • [2026-06-26] Evo-RL: Open Real-World Offline RL on SO-101 and AgileX PiPER: Evo-RL ships an open-source LeRobot-based reproduction of the RECAP iterated-offline-RL loop (π*0.6: a VLA That Learns From Experience (RECAP)) on SO-101 and AgileX PiPER hardware — concrete proof point that the advantage-conditioning policy-extraction trick (binarized text-indicator + supervised likelihood) is portable across robot platforms without per-token log-likelihoods.
  • [2026-06-25] OpenThoughts-Agent: Data Recipes for Agentic Models: OpenThoughts-Agent extends the SFT-only-with-good-data position from OpenThoughts3 reasoning into the agentic regime — Qwen3-32B fine-tune on 100K curated examples reaches 44.8% across 7 agentic benchmarks (+3.9 pp over Nemotron-Terminal-32B). Worth contrasting with the RL-based TMax recipe (TMax: A Simple Recipe for Terminal Agents) which targets overlapping benchmarks — SFT-data-curation vs RL-environment-scaling for agentic capability is now empirically testable at matched base/scale.
  • [2026-06-24] Qwen-AgentWorld: Language World Models for General Agents: Qwen-AgentWorld applies the three-stage CPT → SFT → RL recipe to a new objective — next-state prediction in agentic environments via long CoT — with a hybrid rubric-and-rule reward design (LLM-judged criteria over state correctness/format/semantic plausibility, combined with rule-based exact-match / structural checks where applicable). Extends the GRPO/RLVR family from end-task correctness to environment-dynamics fidelity, and reports that this simulation-fidelity training also serves as a warm-up improving downstream agent performance across 7 benchmarks.
  • [2026-06-23] Seed2.1: A Next-Generation Agent for Real-World Productivity: Seed2.1 reports BeyondAIME 87.0 (Pro) / 88.0 (Turbo) — Turbo above Pro, an unusual inversion — trailing GPT-5.5 91.0 and Gemini 3.1 Pro 90.0; no training-recipe details disclosed in the release page.
  • [2026-06-19] ConTrack: Constrained Hand Motion Tracking with Adaptive Trade-off Control: ConTrack uses a dual-variable update to enforce object-tracking as a hard constraint while letting motion/style fidelity be the secondary objective — a constrained-RL recipe for dexterous manipulation that eliminates per-sequence reward weight tuning (distinct from the policy-gradient-on-text-rewards lineage, but worth noting as the closed-loop-control analog).
  • [2026-06-19] Planning with the Views (ViewSuite): ViewSuite is a clean failure case for the GRPO-as-default frame: at ~2.5% base success, direct PPO plateaus at 3.2%, reward-variance-filtered GRPO at 5.2%, success-only PPO+SFT bootstrapping at 6.2%. The 47.8% breakthrough comes from compressing every rollout (including failures) into a shared view graph and reformulating arbitrary paths into supervised (initial, target, action chain) demonstrations — a graph-of-experience SFT primitive distinct from Training-Free GRPO’s natural-language experience library and from on-policy distillation. Adds a third “RL alternative under near-zero reward” to the open question on this page.
  • [2026-06-18] Robot Learning 2026 Main Lectures — ETH Zürich (Oier Mees): ETH Zürich Robot Learning 2026 main-lectures playlist (Oier Mees) — the imitation-learning + RL + policy-optimization fundamentals portion of the syllabus.
  • [2026-06-16] Qwen-VLA: Unifying Vision-Language-Action Modeling across Tasks, Environments, and Robot Embodiments: Qwen-VLA’s Stage IV runs PPO with sparse binary success rewards on top of a multi-task SFT checkpoint in a single simulation environment (SimplerEnv) and claims the gains transfer to unseen environments and embodiments. Counter-recipe to π*0.6/RECAP’s advantage-conditioning workaround for flow-matching policies’ missing log-likelihoods — Qwen-VLA falls back to plain PPO and absorbs the resulting bias-variance cost.
  • [2026-06-13] InterleaveThinker: Reinforcing Agentic Interleaved Generation: InterleaveThinker extends EditThinker’s GRPO Critique-Refine-Repeat recipe from single-turn image editing to multi-step interleaved generation; introduces an accuracy + step-wise reward design so that single-step GRPO can guide a trajectory with >25 generator calls — directly transferable engineering primitive for long-horizon agentic RL with expensive per-step environment calls.
  • [2026-06-11] Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models: Embodied-R1.5 trains an 8B unified embodied VLM with a multi-task balanced RL recipe whose output-type-specific rewards explicitly target gradient interference across heterogeneous task heads (cognition, planning, correction, pointing) — a concrete recipe choice in the multi-task RL post-training space.
  • [2026-06-10] Test-Time Gradient Guidance of Flow Policies in Reinforcement Learning: QGF (Q-Guided Flow) is the first filed test-time-only RL recipe for flow/diffusion control policies — pre-train a flow policy with vanilla BC and a value-function critic separately, then use the value gradient (with the Jacobian of the denoising chain approximated by identity) to steer the reference policy at inference. Sidesteps both BPTT-through-denoising instability and the DDPO/DPPO/Flow-GRPO horizon blowup that FPO (Flow Matching Policy Gradients) and HuDA already navigate; closest cousin is RECAP’s CFGRL trick (π*0.6: a VLA That Learns From Experience (RECAP)), but QGF moves the entire “improvement” step from training to inference. Reports beating prior test-time RL methods on offline-RL and goal-conditioned offline-RL benchmarks with high-dimensional actions, competitive with training-time SOTA, and favorable model-size scaling precisely because there’s no actor-critic instability.
  • [2026-06-09] LeapAlign: Post-Training Flow Matching Models at Any Generation Step by Building Two-Step Trajectories: LeapAlign is an explicit pushback against the “GRPO-as-default” framing for differentiable samplers — a direct-gradient (no RL) flow-matching post-training recipe that backprops reward through a two-step “leap trajectory” with α-discounted nested gradients, reaching any timestep with constant memory. On FLUX.1-dev with HPSv2.1 it beats DanceGRPO, MixGRPO, ReFL, DRaFT-LV, and DRTune across all six preference metrics and on GenEval (0.7420 vs MixGRPO 0.7232), with the largest gains on layout-driven categories (Position, Attribute Binding) — direct evidence that the wins come specifically from early-step training that policy-gradient methods don’t reach efficiently. Sharpens the open question with Finite Difference Flow Optimization (FDFO): RL Post-Training for Flow-Based Image Generators: what’s the right credit-assignment unit for flow-matching post-training — one image-space gradient copied uniformly (FDFO), backprop through two leaps with nested-gradient discounting (LeapAlign), or SDE-reformulated MDP with group advantages (DanceGRPO/MixGRPO)?
  • [2026-06-04] NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4: Nemotron 3 Ultra uses a four-stage SFT → async-GRPO multi-environment RL → RLHF → Multi-Domain On-Policy Distillation (MOPD) post-training recipe; async RL fully decouples training from inference across separate GPU devices using in-flight weight updates plus MTP-accelerated rollout.
  • [2026-06-02] Stable-Layers: Fine-Tuning Image Layer Decomposition Models with VLM-Scored Reinforcement Learning: Stable-Layers applies Flow-GRPO with LoRA to a pretrained image-layer decomposer (Qwen-Image-Layered), using only VLM-scored rewards. Notes the practical problem that isolated per-sample VLM scores compress into a narrow band, killing GRPO group-relative advantages, and introduces a side-by-side grid-calibration pass to restore variance.
  • [2026-06-02] MAI-Thinking-1: Building a Hill-Climbing Machine: MAI-Thinking-1 (35B/1T MoE) runs three domain-specialist RL climbs (STEM, agentic coding, helpfulness/safety) from a base with no prior reasoning-trace exposure, sustains log-linear gains over thousands of steps via a self-distillation + robust-recipe loop, and consolidates the specialists into one model; reaches 97.0 AIME 25 / 87.7 LiveCodeBench v6 / 52.8 SWE-Bench Pro.
  • [2026-05-30] slime adds lossless delta weight sync for Megatron ↔ SGLang RL disaggregation (PR #1806): slime (the OSS RL framework behind GLM-4.5 → GLM-5.1) adds lossless delta weight sync with disk and NCCL transports for Megatron ↔ SGLang disaggregation (PR #1806) — the train→rollout weight-shipping primitive that closes the loop with AReaL-style async rollout staleness bounds. Motivated by FireworksAI and Cursor.
  • [2026-05-30] Mamoda2.5: Enhancing Unified Multimodal Model with DiT-MoE: Mamoda2.5 uses RL jointly with distillation (rather than after it) when compressing a 30-step diffusion video editor to a 4-step student — an applied data point for combining RL-style preference signals with distillation in a generative-model post-training pipeline.
  • [2026-05-24] History May Repeat Itself: RSI Seen from a Previous AI Era: Yuandong Tian’s ICLR 2026 RSI workshop talk synthesizes the AlphaZero-era model+harness self-improvement loop with the current LLM-RSI agenda, and explicitly raises whether RL is the right primitive at all — pointing at Training-Free GRPO and AlphaEvolve as alternatives, plus LaMCTS (learn the action-space partition before searching it) as a search-side recipe orthogonal to the policy-gradient one. Adds the “RL-vs-not” framing to this page, plus an explicit “harness still matters even with a stronger base” datapoint (23M-param grandmaster-level chess transformer needs search to beat itself at blitz).
  • [2026-05-22] Kimi K2.5: Visual Agentic Intelligence: Kimi K2.5 introduces Parallel-Agent Reinforcement Learning (PARL) — GRPO-family RL on a multi-agent orchestrator policy. Two new conceptual contributions to this page: (a) named failure modes for parallel-agent RL (serial collapse, spurious parallelism) with targeted reward terms for each, plus a staged-reward schedule shifting weight from parallelism to task success across training; (b) a latency-oriented evaluation metric (Critical Steps, the longest path through the orchestrator’s parallel execution graph) that replaces total step count and structurally rewards parallel strategies. Reported result: up to 4.5× Critical-Steps reduction and ~80% wall-clock reduction over single-agent execution on wide-search scenarios; +3.5 absolute BrowseComp points over single-agent context-managed inference (74.9 → 78.4). First filed example of latency entering the RL objective on this page.
  • [2026-05-22] Training-Free Group Relative Policy Optimization: Training-Free GRPO — same group-relative-advantage primitive as vanilla GRPO, but the “policy update” is an Add/Delete/Modify/Keep edit to a natural-language experience library that conditions subsequent rollouts; frozen DeepSeek-V3.1-Terminus + 3 steps × 100 samples matches/beats 32B GRPO-fine-tuned baselines on AIME24/25 and WebWalkerQA at ~500× lower cost. Ground-truth-free variant still helps. Fails when base model is too weak (QwQ-32B regresses).
  • [2026-05-22] Human detectors are surprisingly powerful reward models: HuDA ports GRPO to a video diffusion policy (Wan 2.1 1.3B/14B), using a zero-shot perception-stack reward (ViTDet “person”-confidence worst-window minimum + BLIP similarity to LLM-generated phase captions) — no learned reward network, no human-annotated preference data. Reaches 73% win-rate over Wan 2.1 14B on hard human-action prompts while preserving prompt faithfulness; KL term dropped following DanceGRPO. Out-of-domain GRPO transfer to animal videos and human-object interactions despite the reward being designed for humans.
  • [2026-05-22] MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling: MiroThinker (8B/30B/72B; v1.5 adds 235B-MoE) is post-trained for deep research with a three-stage SFT → DPO + auxiliary SFT loss → online GRPO pipeline. The online GRPO stage uses a streaming rollout queue to handle the long-tailed completion times of multi-hundred-step ReAct trajectories, with a correctness + format-penalty reward and explicit trajectory curation that filters both noisy-correct (repeated API failures) and trivially-incorrect (format failures, action loops) samples. Reaches 47.1% on BrowseComp and 81.9% on GAIA-Text-Only at 72B.
  • [2026-05-22] Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning: Trains a 7B video MLLM to autonomously emit <grounding> and <caption> tags inside CoT via SFT on 10K curated traces + GRPO with correctness+format reward; SFT-only ablation underperforms the base model, isolating the GRPO stage as the source of generalization (+11–18 points OOD over Video-R1-7B).
  • [2026-05-22] vlm-gym: RL gym for vision language models (JAX): Ports GRPO to a VLM (Qwen3-VL-4B in JAX) on GeoGuessr-style street-view localization with a four-level hierarchical reward (country/region/city/coords) using geodesic distance + exponential decay and a coarse→fine weight schedule.
  • [2026-05-22] Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing: InfTool — three-agent role-playing pipeline synthesizes tool-use trajectories with no human labels and trains a 32B base with GRPO + gated rewards in a closed data→model→data loop, reaching 70.9% on BFCL from 19.8% baseline.
  • [2026-05-22] IQuest-Coder-V1 Model Family: Splits post-training into Instruct (instruction-following) and Thinking (reasoning-driven RL) tracks at 7B/14B/40B; the 40B-Loop-Thinking research prototype tests whether looped-transformer structural CoT and reasoning-RL procedural CoT are complementary.

Open questions

Section titled “Open questions”
  • How much of the gain on closed-loop self-synthesis (Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing) comes from GRPO vs. the data loop itself? No filed paper isolates the contribution.
  • Does the hierarchical-reward recipe in vlm-gym: RL gym for vision language models (JAX) transfer to generative VLM tasks (e.g. layout → pose → pixel for image generation), or is it specific to tasks with a natural nested target?
  • Are looped/recurrent compute (structural CoT) and reasoning-RL (procedural CoT) actually additive, redundant, or substitutes? IQuest-Coder-V1 Model Family poses this question via the 40B-Loop-Thinking checkpoint but does not answer it with controlled ablations in the filed material.
  • Self-generated training data risks overfitting to the synthesizer’s quirks; BFCL numbers in Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing are on the same distribution as the synthesizer — how do these models generalize off-distribution?
  • Is the “SFT alone hurts, GRPO alone is unstable” story in Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning a generic finding for structured-CoT formats, or specific to the grounding/captioning tag schema? No filed paper tests dropping SFT for this kind of multi-tag CoT.
  • Is “interaction depth” (MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling §1) a genuinely independent scaling axis, or a re-parameterization of context length and model capacity? The MiroThinker paper shows monotone improvement up to ~600 tool calls but does not isolate the contribution of interaction-budget from the co-scaled context window and base model size.
  • Why does a human-detector-based reward improve animal and human-object-interaction generation as well (Human detectors are surprisingly powerful reward models §4.4)? Two hypotheses: (a) the detector’s “person” confidence is a generic non-rigid-body plausibility prior; (b) GRPO with any well-correlated reward improves overall sample quality. No filed paper distinguishes these.
  • Does the no-KL, on-policy GRPO recipe (Human detectors are surprisingly powerful reward models §3.3, following DanceGRPO) cause reward hacking under longer training or on policies further from the base? The Wan 2.1 results are in a short-training regime where divergence may not have shown up yet.
  • At what base-model capability does Training-Free GRPO transition from “capability amplifier” to “no-op or harmful”? Training-Free Group Relative Policy Optimization shows DeepSeek-V3.1-Terminus benefits, Qwen2.5-72B-Instruct barely benefits, and QwQ-32B regresses on WebWalkerQA — but the threshold is not characterized. Related: is the cliff about raw model size, instruction-tuning quality, native tool-use proficiency, or in-context-learning bandwidth?
  • Is “context-space RL” (Training-Free GRPO) closing all the gap parameter-space RL claims to close, or only the in-distribution gap? Training-Free Group Relative Policy Optimization reports strong cross-domain transfer when swapping the experience library on a frozen model, but does not test compositional regimes (e.g. math + web in a single library, or zero-experience domains the LLM was never given samples for).
  • Do PARL’s named failure modes (serial collapse, spurious parallelism) and the Critical-Steps latency metric (Kimi K2.5: Visual Agentic Intelligence §PARL) generalize beyond K2.5’s specific recipe and reward weighting? PARL is the only filed parallel-agent RL recipe — failure modes and metric design need replication.
  • Is latency-in-reward (Critical Steps) a transferable primitive — e.g. for training video diffusion policies under inference-step budgets, or VLM agents under tool-call cost budgets? The motivation is general but the only existing application is to multi-agent orchestration.
  • If RL is not the right RSI primitive (History May Repeat Itself: RSI Seen from a Previous AI Era), what is the head-to-head comparison? AlphaEvolve (LLM-driven program-search agent) and Training-Free GRPO (context-space updates) span very different design axes; no filed paper benchmarks AlphaEvolve directly, and Training-Free GRPO’s “capability amplifier” caveat makes it a partial substitute at best. The “model + harness” framing the talk borrows from AlphaZero is the closest existing rubric for comparing these.

Papers

Section titled “Papers”

2026-11

Section titled “2026-11”

2026-05

Section titled “2026-05”

2026-04

Section titled “2026-04”

2026-03

Section titled “2026-03”

2026-02

Section titled “2026-02”

2026-01

Section titled “2026-01”

2025-12

Section titled “2025-12”

2025-11

Section titled “2025-11”

2025-10

Section titled “2025-10”

2025-09

Section titled “2025-09”

2025-08

Section titled “2025-08”

2025-07

Section titled “2025-07”

2025-06

Section titled “2025-06”

2025-05

Section titled “2025-05”

Unknown date

Section titled “Unknown date”