SETA: Scaling Environments for Terminal Agents (env repo)

TL;DR

SETA (CAMEL-AI / Eigent.AI / SambaNova, Jan 2026) is a stack for building, training, and evaluating LLM agents that operate inside a real Linux terminal. The seta-env repo specifically ships the Terminal-Bench-compatible synthetic RL environment: an initial 400-task dataset (260 used for Qwen3-8B RLVR) auto-generated from ~1,300 seed Q&A pairs by a two-agent factory (Idea Generation + Datapoint Creation, both driven by Claude Sonnet 4.5) and verified by running Terminal-Bench’s Oracle agent against synthesized solution scripts. Paired with the sibling camel-ai/seta agent repo, it produced SOTA on Terminal-Bench 2.0 (Claude Sonnet 4.5, 46.5%) and Terminal-Bench 1.0 (GPT-4.1, 35.0%), plus a +20.2% unit-test-pass-ratio lift on Qwen3-8B from RL on the synthetic set. The relevance to Luma: this is a working blueprint for “scrape Q&A → synthesize Docker-based verifiable tasks → train via GRPO on remote-Docker RL fleet” applied to a domain (terminal) that has the same verifier-cheap, action-expensive structure as graphics tooling and pipeline-automation tasks.

Key claims

The synthetic dataset is generated by a two-stage agent pipeline: an Idea Generation Agent transforms raw Q&A pairs into a draft_spec.md with RL constraints, then a Datapoint Creation Agent emits the full Terminal-Bench-compatible task (Dockerfile, task.yaml, solution.yaml/.sh, run-tests.sh, tests/) [README; blog “Data Generation Pipeline”].
Validation reuses Terminal-Bench’s own harness: each synthesized task is run with the Oracle agent (tb run --agent "oracle" --dataset-path ./path --task-id x) to confirm the supplied solution actually passes the unit tests in that container [blog “Terminal Environment Validation Pipeline”].
Reward for RL is the unit-test pass ratio per rollout plus a +1 bonus for full success — chosen explicitly because a binary 0/1 reward is too sparse for GRPO advantage estimation on long-tail terminal tasks [blog “Reward Function Design”].
Training is GRPO with Qwen3-8B as base, run inside the AReaL framework, with the CAMEL agent design also demonstrated for rLLM integration; CAMEL agents support fully async model requests and tool calls for efficient rollout [blog “Training Recipe”].
Terminal containers are decoupled from the training host onto an AWS server fleet via a Remote Environment Manager, because running 256+ concurrent Docker containers locally causes severe CPU/IO bottlenecks [blog “Decoupled Terminal and Training Environment”].
Dynamic filtering during training skips tasks where all rollouts get the same reward (no learning signal), focusing optimization on tasks with partial success [blog “Overcoming Challenges”].
A retry loop on tool-call format errors (e.g. malformed JSON from Qwen3-8B) feeds the error message back as tool output rather than aborting, which doubles as an in-loop instruction-following curriculum [blog “Overcoming Challenges”].
Headline benchmark results: 46.5% on Terminal-Bench 2.0 with Claude Sonnet 4.5 (#1, +3% over 2nd); 35.0% on Terminal-Bench 1.0 with GPT-4.1 (#1, +4.7% over 2nd); 3.4% on Terminal-Bench 2.0 with Qwen3-8B baseline; RL-trained Qwen3-8B solves 4 new tasks vs. baseline and lifts averaged unit-test pass ratio 0.38 → 0.45 (+20.2%) across 79 evaluation tasks [blog “Harness Performance” + “Preliminary Terminal Agent RL results”].
The CAMEL terminal toolkit exposes distinct primitives for blocking vs non-blocking shell execution (shell_exec), direct file writes (shell_write_content_to_file), and interaction with running processes (shell_write_to_process, shell_view, shell_wait, shell_kill_process) — the file-write primitive was found especially important for Qwen3-8B-scale models that struggle with nested-quote echo constructions [blog “Terminal Toolkit”].
A separate note-taking toolkit (create_note / append_note / read_note / list_note) is paired with the terminal toolkit as persistent external memory for multi-step tasks — framed as making memory a “first-class, queryable resource” rather than relying on context window alone [blog “Note-Taking Toolkit”].
Failure analysis on Claude Sonnet 4.5 + CAMEL on Terminal-Bench 2.0: successful tasks averaged 79.8 tool calls vs. 128.8 for failures (one pathological case used 452 calls on a metacircular evaluator); strong in Git ops (80%), DevOps (100%), code security (100%); weak in cryptography (33%), low-level compilation (33%), polyglot programming (0%), advanced ML infra (0%), bioinformatics (0%), video processing (0%); ~37% of complete-failure tasks are pass@5=0, of which roughly a third are attributed to absent external-search capability (no Browser Toolkit in this experiment) [blog “When the Agent Fails”].
The synthetic dataset is medium-difficulty by design: across 7 categories (software-engineering, system-administration, security, debugging, systems-programming, networking, devops), only one task was labelled “easy” — the Idea-Gen agent is prompted toward technical-diversity-and-difficulty rather than uniform sampling [blog “Synthetic Dataset Categorization”].

Method

The artifact pasted in Slack is the environments half of the SETA system. The pipeline has two physical repos: camel-ai/seta-env (this one — the synthetic Terminal-Bench-compatible task dataset and download/convert utilities) and camel-ai/seta (the agent harness and RL training code). The synthesis side runs a two-agent factory on top of Claude Sonnet 4.5 via the Claude Agent SDK: the first agent (Idea Generation) rewrites a scraped terminal Q&A into a structured spec with RL-training constraints (technical diversity, language coverage, tool variety, problem-type mix, system complexity), and the second agent (Datapoint Creation) materializes the spec into a Terminal-Bench task bundle (Dockerfile, task.yaml, solution script, run-tests.sh). A validation pipeline then invokes Terminal-Bench’s Oracle agent inside the synthesized container to confirm the included solution actually passes the included tests.

The training side uses CAMEL’s ChatAgent integrated with AReaL for GRPO RL on Qwen3-8B. The agent issues async multi-turn tool calls against the CAMEL Terminal Toolkit (which distinguishes blocking from non-blocking shell, and exposes interactive-process I/O explicitly) plus the Note-Taking Toolkit (persistent external memory). Each rollout requires its own Docker container; at 256+ concurrent rollouts the system offloads container lifecycle management to an AWS fleet through a Remote Environment Manager exposing containers over TCP. Reward is dense (unit-test pass ratio + completion bonus), and dynamic filtering drops zero-variance tasks each step.

Results

On evaluation (no RL, frontier models): 46.5% on Terminal-Bench 2.0 with Claude Sonnet 4.5 + CAMEL harness (SOTA among Sonnet-4.5 agents, +3% over 2nd); 35.0% on Terminal-Bench 1.0 with GPT-4.1 + CAMEL (SOTA among GPT-4.1 agents, +4.7% over 2nd). Baseline Qwen3-8B with CAMEL harness sits at 3.4% on TB-2.0. After RL on a 260-task subset of the synthetic 400-task dataset (10 epochs, GRPO, AReaL): the trained Qwen3-8B model solves 4 new tasks (i.e. tasks the baseline never solved across 8 runs) and lifts averaged unit-test pass ratio from 0.38 → 0.45 across the 79-task filtered eval split — a +20.2% relative improvement. The Terminus 2 / OpenThinker-Agent-v1-SFT baseline scores 0%.

The failure-mode breakdown is more interesting than the headline numbers for anyone thinking about applying this recipe: the Claude Sonnet 4.5 harness loses essentially all bioinformatics (0/3), video processing (0/2), polyglot programming (0/2), and advanced ML infrastructure (0/4) tasks. The blog attributes ~1/3 of complete failures to the absence of a Browser Toolkit (i.e. the model lacks an external search/docs lookup channel) and flags adding it as the next planned mitigation.

Why it’s interesting

For Luma, the value isn’t terminal agents per se — it’s the scalable-environment-synthesis + verifier-cheap-RL recipe demonstrated end-to-end. SETA is the closest filed analogue to “scrape domain Q&A → auto-generate Dockerized verifiable tasks → fleet-out the Docker, GRPO with a dense-test-pass reward, lift a small open model by +20%”. The same shape applies cleanly to graphics-pipeline tasks (Houdini node graphs, Blender scenes, FFmpeg pipelines) or 3D-tool agents where success is unit-testable but actions are expensive and stateful.

Two specific design choices are worth carrying forward. First, decoupling environment containers from the training host onto a remote fleet is the kind of systems detail that’s invisible until you try to run 256-concurrent Docker locally and fall over — this matches the streaming-rollout pattern from MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling §5.3 (long-horizon RL where rollouts are heavily long-tailed and need infrastructure, not just an objective). Second, using Terminal-Bench’s own Oracle agent as the synthetic-task verifier is a very clean closure: the synthesizer doesn’t need its own ground-truth pipeline, it parasitizes the benchmark’s existing harness. This is the same trick Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing uses (verified-trajectory gates) and a generalizable pattern for any domain with a pre-existing scorer.

The contrast with sibling tool-use papers is that SETA’s loop is not fully self-play — the synthesizer (Claude Sonnet 4.5) is a separate, stronger model than the trained student (Qwen3-8B). That sidesteps the “synthesizer-and-verifier-from-same-model-family” reward-hacking risk flagged on Synthetic Training Data and Reasoning RL, but at the cost of a frontier-API dependency in the data pipeline.