Skip to content
Bud's Wiki

RL Environment Platforms

TL;DR

Section titled “TL;DR”

Across the recent filings, “RL environment” has become a packaged, distributable artifact — a Docker-bootstrapped backend + per-task verifier + standardized agent-interaction interface — rather than a per-project research scaffold. The three filed instances span a clear scale gradient: SETA ships ~400 Terminal-Bench-compatible tasks as a single-repo synthesizer, Toolathlon-GYM ships 503 MCP/PostgreSQL enterprise-workflow tasks as a Docker Compose stack, and OpenReward ships 330+ environments × 4.5M+ tasks behind a managed API governed by the Open Reward Standard (ORS, an MCP extension). The unifying bet: standardize the env↔trainer interface, decouple env compute from training compute, and the “scrape-domain-Q&A → auto-generate verifiable tasks → GRPO” pipeline becomes plug-and-play across domains.

Key claims

Section titled “Key claims”

Recent contributions

Section titled “Recent contributions”
  • [2026-07-20] Danfei Xu — two root-level paradigm shifts in robot learning: Sim2Real for locomotion, Behavior Cloning for manipulation: Danfei Xu casts Sim2Real (for locomotion) as one of only two root-level paradigm shifts of the deep-learning era in robot learning — an implicit endorsement of simulator platforms as the load-bearing substrate for that sub-field.
  • [2026-07-16] Schema — Frontier Models with Our Harness Achieve ~99% on ARC-AGI-3 Public: Schema is the first filed harness result on ARC-AGI-3 (the concept’s most demanding filed environment): self-reported 98.98% Public RHAE with Opus 4.8 / Fable 5 fallback versus 42.83% for the same pair under unconstrained Claude Code, isolating a 56.15% harness effect. Sharpens the verifier-cheap / action-expensive design pattern: here the learned world-model program is the verifier and BFS inside it substitutes for environment actions; residual failures are 5-6 games where representation discovery, not verification, is the bottleneck.
  • [2026-07-15] Pantograph introduces Pan-1 — Minecraft model with RL-based pretraining, arguing video games are the right testbed for robotics: Pantograph explicitly argues Minecraft-class video games are the right RL testbed for robotics methods (reproducible / safe / open-ended / long-horizon) — treating game environments as first-class robotics-adjacent RL platforms rather than just eval curiosities. Product-tweet framing; Pan-1 launch is the first datapoint attached to this argument on the wiki.
  • [2026-07-14] Tactile Genesis: Exploring Tactile Sensors at Scale for Learning Dexterous Tasks: Tactile Genesis (Chung et al.; NSF/DARPA/Amazon/CMU/Google-supported) extends the env-as-package pattern with a tactile front-end integrated into Genesis World — 8 sensor abstractions (contact / depth / kinematic force-torque / elastomer / proximity / temperature / audio) under one interface, 20,000+ parallel envs and 1,000+ taxels on a single GPU, 3-20× throughput over TacSL / Tacmap / HydroShear / FOTS. Turns “sensor hardware ablation” from a hardware-swap-per-lab problem into an in-simulator sweep, and validates the resulting placement-over-type design directive with sim-to-real transfer on a real XHand1.
  • [2026-07-14] How Claude Performs on Robotics Tasks: Anthropic assembles a mixed sim-and-real evaluation harness on top of Mujoco (classic control + Go2/G1 locomotion), LIBERO kitchen scenes (direct + VLA-supervised manipulation), and a physical Unitree Go2 for real-world navigation. Introduces TwinFlipper as a novel chaotic-dynamics classic-control task explicitly designed to avoid pretraining-corpus contamination (unlike inverse pendulum and hopper), and an eleven-task Go2 navigation suite with self-monitoring probes (drift_detection: notice commands are being silently corrupted; explore_report: answer layout questions from memory). Sim runs are paused between LLM calls to approximate an upper bound on capability by removing real-time latency — the paper quantifies the gap: ~83 Hz needed vs 0.2–0.4 Hz non-reasoning inference, roughly two orders of magnitude. Complements the packaged-env pattern (SETA, Toolathlon-GYM, RoboCasa365, RoboDojo) with a frontier-lab safety-evaluation framing where the environment is instrumented specifically to measure LLM-as-controller capability trends across model generations.
  • [2026-07-14] GaP: A Graph-as-Policy Multi-Agent Self-Learning Harness For Variational Automation Tasks: GaP (Berkeley/NVIDIA/CMU; Goldberg, Fan, Zhu) extends the CaP-Gym env-as-package pattern with a parameterized simulator that spins up per-candidate-graph instances for parallel rehearsal, with pre/post-node state and contact feedback logged at every node boundary for failure localization. Ships 8 new Variational Automation benchmarks (4 sim, 4 real — grocery fulfillment, grocery packing, popcorn, USB-C insertion, bimanual crate washing) explicitly designed with larger pose/geometry variation than LIBERO/LIBERO-Pro; the sim table pairs each benchmark with per-instance randomization of object poses and arrangements.
  • [2026-07-14] ABot-AgentOS: A General Robotic Agent OS with Lifelong Multi-modal Memory: EmbodiedWorldBench (companion benchmark to ABot-AgentOS, Alibaba): 16 indoor/outdoor/hybrid scenes × 4 difficulty levels × 200+ tasks spanning navigation, object search, NPC dialogue, dynamic events, with trace-grounded scoring rather than terminal-state scoring — targeted at long-horizon embodied agent OS evaluation.
  • [2026-07-09] RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies: RoboDojo (HKU MMLAB + Berkeley) unifies the sim and real halves of the env-platform pattern: 42 Isaac Sim tasks partitioned across 5 capability dimensions (generalization, memory, precision, long-horizon, open-vocab) plus 18 real tasks on cloud-remote standardized hardware (RoboDojo-RealEval), with XPolicyLab as a common policy-integration layer; ships a public leaderboard from a 30-policy sweep — the largest filed policy pool on a single manipulation benchmark, complementing RoboArena’s real-only crowd-sourced protocol and RoboCasa365’s sim-only kitchen suite.
  • [2026-07-05] Agent² RL-Bench: Can LLM Agents Engineer Agentic RL Post-Training?: Uses ALFWorld / WebShop / DeepSearchQA as packaged environment substrates but flips the evaluated agent: instead of measuring an RL-trained policy inside the env (the concept’s usual pattern), it measures whether a CLI coding agent can engineer the training pipeline (data, rewards, rollout collection, GRPO/DPO/PPO code) that would produce such a policy. Behavioral instrumentation records every submission and code revision inside the workspace — adds a “how the agent constructed the pipeline” observability layer on top of the standard verifier-cheap / action-expensive env-as-package design.
  • [2026-07-04] RoboWorld: Fast and Reliable Neural Simulators for Generalist Robot Policy Evaluation: RoboWorld turns a single real initial frame into an interactive neural simulator and runs closed-loop VLA policy evaluation with no simulator engineering or asset creation, including eight extreme unseen-scene variants of RoboArena episodes.
  • [2026-07-01] Scale Robot Policy Evaluation with Ray (Distributed Sim-Eval on Anyscale): Anyscale’s engineering post lays out a Ray + Anyscale reference architecture for large-scale robot-policy evaluation: policy (GR00T-N1.7-3B) as an autoscaling Ray Serve deployment on one GPU pool, Isaac Lab simulators as isolated Ray tasks on a separate GPU pool, HTTP boundary between them, action-chunk amortization (action_horizon = 8 steps per ~1.6 s inference) across the hop; standardizes the rollout runtime the way SETA/Toolathlon-GYM/OpenReward standardize the task side and RoboArena/RoboCasa365/Genesis standardize the evaluation substrate — same verifier-cheap/action-expensive design principle where action cost is GPU physics + rendering rather than Docker sandbox setup.
  • [2026-06-30] ASPIRE — self-evolving sensorimotor skill library that compounds across robots and the sim-to-real gap (Jim Fan / NVIDIA GEAR): Jim Fan announces NVIDIA GEAR’s ASPIRE — a sim+real co-training setup where coding agents run evolutionary search over control programs against multimodal sensory traces from both simulated and real robots, with successful programs distilled into a self-evolving code skill library. Full stack promised as open-source alongside a gallery of 150+ tasks and 90+ self-taught skills.
  • [2026-06-29] CHORD: Object-Centric Contact Wrench Guidance for Dexterous Manipulation: CHORD ships a 4,739-task simulation-ready bimanual dexterous manipulation benchmark constructed from mocap datasets and reconstructed in-house human videos, paired with the demonstrations themselves — the largest filed dexterous-RL task surface and a direct answer to the “where do dexterous RL environments come from at scale” gap.
  • [2026-06-29] SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation: SimFoundry reconstructs deployable simulation environments from a single video, with cousin-style automated augmentation along object / scene / task axes — a user-supplied real scene → on-demand parameterized sim environment pattern complementing RoboCasa’s fixed kitchen-task benchmark and Genesis World’s general-purpose simulator.
  • [2026-06-24] Qwen-AgentWorld: Language World Models for General Agents: Qwen-AgentWorld is the strongest filed counter-bet to the env-as-package design pattern: instead of shipping environments as Docker stacks (SETA, Toolathlon-GYM) or managed services (OpenReward) or synthesizing executable SQL-backed environments (AWM), train a single MoE LLM (35B-A3B / 397B-A17B) to simulate environment dynamics across 7 agentic domains via long CoT reasoning. Reports that this simulator scales to thousands of environments controllably and yields agentic-RL gains exceeding real-environment training alone. Inverts the verifier-cheap / action-expensive assumption: when the simulator is a learned model, both verifier and action become cheap inference passes.
  • [2026-06-20] Playful Agentic Robot Learning: RATs reuses LIBERO-PRO and MolmoSpaces (the same env-as-package artifacts CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation benchmarks against) but flips the role: the env is now a play substrate for skill-library acquisition rather than a downstream evaluator. Verifier-cheap + action-expensive design extends to a setting where rollouts are exploratory and the verifier scores intermediate progress, not just final task success — adds a new use-case axis (skill discovery) alongside policy-fitting and evaluation.
  • [2026-06-17] ABC: Scalable Behavior Cloning with Open Data, Training, and Evaluation: ABC Sim extends the env-as-evaluation-substrate pattern (Genesis World, RoboCasa365) to bimanual manipulation: MuJoCo physics + an offline Blender re-rendering pipeline for higher-fidelity ray-traced frames, 400h of sim-teleop across 20 tasks. Headline contribution to this cluster is a quantified sim-real correlation study across 12 paired checkpoints (r=0.85 strict success, r=0.91 task progress) — the second filed datapoint after Genesis (Pearson 0.8996) supporting the “use sim purely as cheap eval, decouple from training” thesis.
  • [2026-06-16] ENPIRE: Agentic Robot Policy Self-Improvement in the Real World: ENPIRE extends the verifier-cheap / action-expensive env-as-package pattern from terminal and enterprise-workflow surfaces to physical manipulation: each task is wrapped as self-resetting (randomized initial state + reset behavior + verification of reset success) and self-verifying (multi-camera detector + segmentation fused into a binary reward), with the auto-reset + multi-camera verifier playing the role the Docker sandbox plays in SETA / Toolathlon-GYM; also supplies simulation evaluation in RoboCasa as the parallel sim track.
  • [2026-06-14] LEGS: Fine-Tuning Teleop-Free VLAs for Humanoid Loco-manipulation in an Embodied Gaussian Splatting World: LEGS is a hybrid MuJoCo + 3DGS simulator with a procedural Walk → Pick → Place motion-primitive generator that emits verified-success-only labeled demonstrations — a humanoid-loco-manipulation analog to RoboCasa365’s auto-trajectory pipeline but with photorealistic 3DGS backgrounds replacing meshed kitchens.
  • [2026-06-14] RoboArena rolls back evaluations after benchmark hacking observed since April (Pranav Atreya announcement): First filed integrity-attack datapoint on any platform in this cluster — RoboArena lead author Pranav Atreya publicly announces that benchmark hacking has been observed on RoboArena since April 2026, that mitigations are now in place, and that affected evaluations (including, per the Slack pointer, NVIDIA’s Cosmos) have been rolled back; the §3 “robustness against single-actor manipulation” property of RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies survived only because the maintainer team detected the attack out-of-band, not because the protocol caught it automatically.
  • [2026-06-08] RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies: RoboArena (CoRL 2025) is the real-world / human-evaluator counterpoint to the standardized env-as-package pattern — instead of a fixed task set with an automatic verifier, evaluation is crowd-sourced across seven academic institutions on the DROID platform, with double-blind pairwise A/B comparisons aggregated by a task-aware Bradley-Terry extension; a 1:1 evaluation-credit system balances supply and demand. At matched episode budget across 612 paired comparisons over 7 generalist DROID policies, the crowd-sourced ranking matches a 4,284-episode oracle ranking more accurately than the conventional centralized 17-task DROID protocol — direct evidence that the “verifier-cheap, action-expensive” design assumption (Docker sandbox + unit tests) breaks for embodied generalist policies where the deeper signal lives in unstandardized real-world conditions a human judge can resolve.
  • [2026-06-08] RoboCasa365 — Large-Scale Simulation of Everyday Tasks for Generalist Robots: RoboCasa365 — ICLR 2026 release of a kitchen-domain manipulation env platform with 365 LLM-authored tasks, 2,500+ scenes, 3,200+ objects, and 2,200+ hours of human+auto demos. First filed manipulation-specific entry in the env-as-package pattern, with built-in Diffusion Policy / pi0 / GR00T baselines and a public leaderboard. The 600h-human + 1,600h-auto demo split mirrors SETA’s frontier-synthesizes-tasks pattern on the trajectory side.
  • [2026-06-03] Project Chrono — Open-Source Multi-Physics Simulation Engine: Project Chrono — long-running open-source multi-physics simulator covering rigid-body, FEM, granular, fluid-solid, and ODE dynamics with a ROS2-integrated camera/LiDAR/GPS/IMU/SPAD sensor suite. Predates the recent wave of pip-installable robotics platforms and is the closest C++ analog to Genesis World.
  • [2026-06-01] Genesis World — Simulation platform for general-purpose robotics & embodied AI learning: Open-source physical-robotics environment platform (Apache 2.0): unified multi-physics scene API + multi-embodiment support (Franka, Go2, humanoid, drones), heterogeneous parallel simulation, broad sensor suite (RGB, depth, lidar, tactile, IMU, contact, temperature). First filed RL env platform whose underlying simulator is itself a research artifact, not a thin wrapper around an existing engine.
  • [2026-06-01] The Role of Simulation in Scalable Robotics, Genesis World 1.0, and the Path Forward: Genesis World 1.0 extends the env-as-package pattern to physical robotics — Apache 2.0 stack (physics + renderer + compiler) framed primarily as the evaluation substrate, with sim-to-real correlation (Pearson 0.8996) as the trust metric and a perturbation-axis taxonomy (visual/behavioral/semantic) replacing scalar success rates. Where the existing terminal/coding env entries are verifier-cheap + action-expensive via Docker, Genesis is verifier-cheap + action-expensive via full multi-physics + path-tracing.
  • [2026-05-27] stable-worldmodel: A Platform for Reproducible World Modeling Research and Evaluation: stable-worldmodel — extends the env-as-package pattern from terminal/coding agents to world-model research: bundled environments with controllable visual / geometric / physical factors of variation, plus a Lance-based data layer with LeRobot ingestion. First filed instance applying the standardized-env+verifier discipline to the WFM evaluation surface rather than to RL training.
  • [2026-05-23] OpenReward — 330+ RL Environments through one API (General Reasoning announcement): OpenReward launches as the first managed-platform datapoint — 330+ environments × 4.5M+ tasks behind an HTTP API, ORS protocol extending MCP with RL primitives, autoscaled sandbox compute provisioned per rollout, trainer compatibility announced for Tinker / Miles / Slime. Turns “env-as-package” into “env-as-managed-service” and shifts the quality-control problem from per-env synthesizer to platform-level policing.
  • [2026-05-23] Toolathlon-GYM: Large-Scale Long-Horizon Environments for Tool-Use Agents: Toolathlon-GYM (Eigent.AI + HKUST-NLP + CAMEL-AI) — 503 long-horizon multi-MCP tasks with brand-obfuscated descriptions, 25 MCP servers, 8.2 MB PostgreSQL dump as shared mock backend. Adds the enterprise-workflow surface (Canvas LMS, Snowflake DW, WooCommerce, Yahoo Finance) to a space that previously held only terminal/shell.
  • [2026-05-22] SETA: Scaling Environments for Terminal Agents (env repo): SETA (CAMEL-AI + Eigent.AI + SambaNova) — ~400 Terminal-Bench-compatible tasks auto-synthesized by a Claude-Sonnet-4.5 two-agent factory and validated by Terminal-Bench’s own Oracle agent. First filed instance of the “env compute on a separate AWS fleet” pattern, and a +20.2% Qwen3-8B GRPO lift on the synthetic 260-task subset.

Open questions

Section titled “Open questions”
  • How does task quality scale with a federated, third-party-authored environment catalog (OpenReward) compared to a curated single-team release (SETA, Toolathlon-GYM)? No filed paper measures the quality-vs-scale curve.
  • Does the MCP/ORS abstraction actually let trainers drop a new env in without per-env special-casing, or does it leak enough that GRPO recipes still need env-specific reward shaping? Toolathlon-GYM ships uniform tools, OpenReward claims protocol-level uniformity, but neither reports a “trainer-agnostic” headline number.
  • What is the right granularity for “task” vs “environment”? OpenReward’s 4.5M tasks / 330 envs ratio (≈13.6K tasks/env) implies envs are parameterized task generators, not single tasks; SETA’s ~400 tasks live inside a single env; Toolathlon-GYM’s 503 tasks live across 25 MCP servers. The dataset-vs-env boundary is unclear in the filed material.
  • Where is the RL-recipe ceiling on this kind of training? SETA reports a +20.2% unit-test-pass-ratio lift on a small base (Qwen3-8B) over a small training set (260 tasks). Toolathlon-GYM is dataset-only at filing time. OpenReward is platform-only. No filed result yet pins down the asymptote.
  • How does the “frontier model synthesizes tasks, smaller model trains on them” recipe (SETA) interact with the “third-party authors author the env, anyone trains” recipe (OpenReward)? The two have different incentive structures for env quality and different attack surfaces for synthesizer-bias.
  • Are brand-obfuscated task descriptions (Toolathlon-GYM) sufficient to prevent shortcut behavior in agents that were pretrained on the un-obfuscated versions (Notion / Google Calendar / Snowflake docs)? No filed paper measures this.
  • Does the verifier-cheap-action-expensive design pattern have natural limits — i.e. is there a domain where verifier compute dominates action compute and these recipes invert? Image/video generation evaluation (where a learned reward model is the dominant cost) is the obvious candidate.

Papers

Section titled “Papers”

2026-05

Section titled “2026-05”

2026-04

Section titled “2026-04”

2026-03

Section titled “2026-03”

2026-01

Section titled “2026-01”

2025-12

Section titled “2025-12”

2025-11

Section titled “2025-11”

2025-02

Section titled “2025-02”