Toolathlon-GYM: Large-Scale Long-Horizon Environments for Tool-Use Agents

TL;DR

Toolathlon-GYM (Eigent.AI, in collaboration with HKUST-NLP’s prior Toolathlon project, CAMEL-AI) is a fully-local, Docker-compose-bootstrapped tool-use environment shipping 503 long-horizon multi-tool tasks, 25 MCP servers, and an 8.2 MB PostgreSQL dump as the shared mock backend (Canvas LMS, Snowflake DW, WooCommerce, Yahoo Finance, YouTube, 12306). Each task is a 4–8-MCP orchestration goal — pull data from a mock enterprise DB, produce an Excel report, schedule a calendar event, send a summary email — automatically verified by a per-task evaluation/main.py against reference workspace ground truth. No external API calls or human graders at run time. This is the bigger, MCP-centric, enterprise-workflow sibling of SETA’s terminal-agent environment (SETA: Scaling Environments for Terminal Agents (env repo)) and a directly trainable substrate for the agentic-RL recipes filed on this wiki.

Key claims

503 long-horizon tasks at MCP-count distribution 4–8 tools per task, with 4–7 being the majority — explicitly chosen to require cross-system coordination rather than single-tool lookups [README §Dataset statistics].
25 MCP servers covering filesystem, Excel, emails, terminal, Snowflake, Canvas, WooCommerce, Yahoo Finance, YouTube, 12306 rail, Google Calendar/Forms, Notion, Word, PowerPoint, scholarly, arxiv-latex, playwright, fetch, pdf-tools, memory, howtocook, etc. [README §MCP server coverage].
Entire backend runs from a single local PostgreSQL instance (toolathlon_gym @ localhost:5432) auto-initialized from db/init.sql.gz (8.2 MB); no external API calls required at evaluation time [README §The mock database].
Mock data is derived from real-world sources: Kaggle OULAD (LMS, 22 courses / 28,865 users / 32,663 enrollments / 173,912 submissions / 77 quizzes), Kaggle HR Analytics + simulated Snowflake DW (50,000 employees / 20,000 sales orders / 31,588 support tickets), Yahoo Finance (50 tickers / 3,510 price records), Kaggle Amazon + DummyJSON (82 products / 150 orders / 50 customers / 396 reviews) [README §The mock database].
Each task is a four-file bundle: task_config.json (allowed MCP servers), docs/task.md + docs/agent_system_prompt.md (description shown to agent), preprocess/main.py (DB state setup before run), evaluation/main.py (automated evaluator against groundtruth_workspace/), with initial_workspace/ pre-loaded files [README §Task structure].
Task descriptions use brand-obfuscated language (“knowledge base” instead of Notion, “shared calendar” instead of Google Calendar, “learning management system” instead of Canvas) — same obfuscation convention as the parent Toolathlon project — to prevent keyword-shortcut behavior and force genuine tool-use reasoning [README §Task structure].
Initial-workspace files span 11 distinct formats: .md (most common, task briefs and methodology guides), .pdf (reference docs / policies the agent must parse), .json (parameterized configuration), .xlsx (pre-filled templates with predefined columns), .csv (tabular reference data to join with DB results), .txt, .py (starter scripts to complete and execute), .pptx, .docx, .bib, .gz [README §Initial Workspace Files].
Concurrent rollouts are supported via run_parallel.sh with a FIFO-based semaphore controlling concurrency; each task gets its own isolated environment (dedicated PostgreSQL + agent container + Docker network) so there is no shared state between concurrent runs [README §Run Tasks in Parallel].
Agent harness ships as a CAMEL-AI example with provider-agnostic plumbing (MODEL_PLATFORM: openai_compatible / openai / anthropic / gemini); per-turn LLM request/response logs and full trajectories are persisted to dumps/<task>/<timestamp>/<provider>_<model>/ [README §Quick Start, §Model Provider Reference].
Built on the format / evaluation framework / MCP interfaces / DB schema design of HKUST-NLP’s Toolathlon (the benchmark); this dataset extends with a substantially larger task pool and bigger mock data for training and evaluation use [README §Acknowledgements].

Method

Toolathlon-GYM is shipped as a Docker Compose stack: toolathlon_pg (PostgreSQL 15, auto-initialized from a versioned SQL dump) plus a toolathlon-pack:latest agent image rebuilt per repo. To run a task, scripts/run_containerized.sh <task-id> spawns a fresh agent container scoped to that task’s task_config.json (which restricts the MCP server whitelist), runs the per-task preprocess/main.py to seed initial DB state, executes the agent under a step budget (default 100), then invokes evaluation/main.py to compare the agent’s workspace/ output against groundtruth_workspace/. For training-scale evaluation, run_parallel.sh N runs up to N tasks concurrently with full isolation (separate PostgreSQL + agent container + Docker network per task) and aggregates per-task results into benchmark_logs/<run>/summary.csv. The MCP-server diversity is the design lever: the dataset is structured so that the most common output paths exercise filesystem + Excel + emails + terminal in combination, while domain-specific servers (snowflake, canvas, woocommerce, playwright_with_chunk, fetch, pdf-tools, memory, youtube-transcript, howtocook) anchor task families with different reasoning bottlenecks.

A task example at the 7-MCP tier (arxiv-research-pipeline-notion-excel, allowed tools = scholarly + arxiv_local + terminal + excel + notion + filesystem) asks the agent to search papers on LLMs / prompt engineering / in-context learning, run a synthesis Python script in the terminal to compute relevance scores and emit structured JSON, populate a three-sheet Excel file (Paper_Catalog / Method_Comparison / Research_Gaps), and produce a Notion page titled “LLM Research Hub” with a research dashboard. The 8-MCP tier (arxiv-research-workflow-pipeline) adds Google Calendar + emails + pdf-tools and a 2,000-word synthesis deliverable. The complexity gradient from 4-MCP to 8-MCP is the dataset’s stress axis on cross-system coordination depth.

Results

The repository is a dataset + harness release; the README does not report fine-tuning results on the 503-task pool. The reference configuration runs evaluation across Claude Sonnet 4.5 (via aihubmix OpenAI-compatible endpoint), GPT-5.2, and Gemini 3 Flash; benchmark-scale results are not enumerated in the README at filing time. The dataset is positioned as the training/eval substrate; the deliverable claim is the scale itself — 503 tasks × 25 MCP servers × ~140,000 underlying mock records is substantially larger and more tool-diverse than the parent Toolathlon benchmark.

Why it’s interesting

For Luma’s research team, this is the MCP-centric, enterprise-workflow analogue of SETA: Scaling Environments for Terminal Agents (env repo) (the closest filed comparable): SETA ships ~400 Terminal-Bench-compatible tasks for terminal-shell agents and uses Claude Sonnet 4.5 as a synthesizer to seed Qwen3-8B RL; Toolathlon-GYM ships 503 tasks with a 25-MCP-server orchestration surface that exercises end-to-end application workflows rather than shell semantics. Both are CAMEL-AI / Eigent.AI artifacts and both target the same RL-from-verifiable-tasks recipe — Toolathlon-GYM is the next-step substrate for the same agentic-RL recipes that Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing and MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling formalized on BFCL / WebWalkerQA. The brand-obfuscated task descriptions are a deliberate guard against the “keyword shortcut” failure mode that Tool-Use Agents flags in its open questions on synthesizer-and-verifier-from-same-model-family bias.

The systems design also lines up with the lesson from SETA’s Remote Environment Manager: isolating PostgreSQL + Docker per concurrent rollout is exactly the pattern needed to avoid the “256-concurrent-Docker locally” failure mode. Where SETA chose terminal as the canonical verifier-cheap-action-expensive domain, Toolathlon-GYM picks structured enterprise data + multi-output deliverables (spreadsheet + email + calendar + doc), which is closer to the kind of multi-modal-deliverable workflows that show up in production pipelines and matches the Agentic Software Engineering benchmark-quadruple expansion (SWE-Bench Pro + Terminal-Bench + OSWorld + GDPval) at the data side.