Toward Scalable Terminal Task Synthesis via Skill Graphs

TL;DR

SkillSynth synthesizes diverse terminal-agent training tasks by walking a scenario-mediated skill graph rather than scaling task count blindly. Skills (Linux command-line workflows) are collected from ClawHub + GitHub, each tagged with a precondition and postcondition scenario; semantically compatible scenarios across skills are merged into nodes, turning skills into directed edges. Inverse-frequency random walks then sample workflow paths that get instantiated by a multi-agent harness (planner + constructor + oracle verifier + rubric judge) into executable tasks with a Docker env, oracle solution, and verification scripts. The framework produces 3,560 verified tasks at 95.7% oracle pass and $27.3/task, and SFT trajectories from these tasks lift Qwen3-32B to 33.8 / 29.6 on Terminal-Bench 1.0 / 2.0 — beating Qwen 3 Coder 480B on TB 2.0. Tasks from this pipeline were used to train Hy3 Preview.

Key claims

• Execution trajectories decompose into (scenario, skill) sequences, and the training objective factorizes over the conditional product of scenarios × admissible skills — making explicit coverage of that space (not raw task count) the right scaling target [§2, Eq. 4].
• The constructed skill graph contains 82,073 deduplicated scenarios, 57,214 filtered skills, and 185,529 LLM-verified bridges; 85.6% of scenarios lie in the largest weakly-connected component, with 16,632,220 enumerable paths of length ≥7 [§4.5].
• Inverse-frequency-weighted random walks on the skill graph (visit counts on scenarios + usage counts on skills, with monotone progression) steer the empirical sampling distribution toward uniform coverage over (scenario × skill), avoiding the high-degree-node trap of uniform walks [§3.3, Algorithm 1].
• Multi-agent harness (planner → constructor → execution-based oracle check + LLM-as-Judge rubric check + multi-turn repair) reaches 95.7% oracle pass and 92.0% combined pass on 3,721 sampled paths; the verify-then-repair loop recovers 721 first-round failures [§4.2, Table 2].
• SkillSynth tasks are harder than baselines: 38% of 3,560 instances are 0/3 unsolved by Hy3 Preview vs 16% for single-skill and 27% for randomly-composed multi-skill seeds; Claude Opus 4.6 needs 37 steps on average [§4.2, §4.4, Table 4].
• Qwen3-32B fine-tuned on SkillSynth trajectories reaches 33.8 / 29.6 on Terminal-Bench 1.0 / 2.0 — outperforming Qwen 3 Coder 480B (23.9 on TB 2.0) and beating single-skill SFT by +8.4 / +8.3 and random multi-skill SFT by +3.0 / +3.8 points [Table 3, Table 4].
• Trajectory diversity gains are measurable: 1,000 trajectories per strategy → SkillSynth tasks yield 31% more unique scenario-skill pairs than single-skill and 19% more than random multi-skill, after embedding + clustering deduplication [§4.5].
• 10,680 trajectories were sampled with MiniMax M2.7 as teacher (3 rollouts per task); both successful and failed trajectories are retained for SFT to preserve coverage of error-recovery patterns [§4.3].
• Dominant agent failure modes on synthesized tasks are partial implementation (42.2%) and over-trust in inline self-tests (29.5%) — both forms of substituting self-narrated verification for specification-grounded testing [§4.6, Table 5].
• Failed rubric checks are dominated (77%) by instruction↔test misalignment (over/under-specification); these tasks are retained for SFT but discarded for RL to avoid spurious reward signal [§4.2].

Method

Three stages. (1) Graph construction: skills from ClawHub and GitHub are filtered for Linux-executability, structured workflow, no jailbreak content, and deterministic verifiability. An LLM infers precondition + postcondition scenarios per skill; scenarios are embedded and hierarchically-agglomeratively clustered (with Louvain coarse bucketing) for semantic deduplication. Cross-skill alignment retrieves top-1,000 most similar pre/postcondition counterparts and uses an LLM judge bidirectionally to confirm compatibility, then merges aligned scenarios into unified nodes. A final LLM filtering pass over (scenario, skill, scenario) triples removes invalid transitions. (2) Path sampling: inverse-frequency random walks where source-node probability ∝ 1/(1+N_s(v)) and edge probability ∝ 1/(1+N_e(s)); visited nodes/edges are blocked within a single walk; paths in [L_min, L_max] = [1, 7] with unseen skill-sets are accepted. (3) Multi-agent harness: planner converts the path into sub-objectives + expected outputs; constructor generates the instruction, initial filesystem, Docker env, oracle solution, and verification scripts; execution-based check runs the oracle in-container while LLM-as-Judge checks instruction↔test alignment and self-containedness. Up to K_max repair cycles × T_max tool calls each.

Results

• 3,721 sampled paths → 3,423 fully passing + 137 oracle-only passing = 3,560 usable tasks (95.7% oracle pass, 92.0% combined). Avg 2.31 repair cycles, 11 tool calls; 721 first-round failures recovered. Avg cost $27.3 per verified task [§4.2].
• Difficulty distribution (Hy3 Preview, 3 attempts): 38% 0/3, 18% 1/3, 19% 2/3, 25% 3/3 [§4.2].
• Terminal-Bench 1.0 / 2.0: Qwen3-8B+SS 17.1 / 13.5, Qwen3-14B+SS 22.9 / 19.9, Qwen3-32B+SS 33.8 / 29.6 — Qwen3-32B+SS beats Qwen 3 Coder 480B (23.9 TB 2.0) and Grok 4 Fast (31.3 TB 1.0) [Table 3].
• Ablation vs same-harness baselines on Qwen3-32B: single-skill 25.4 / 21.3, random multi-skill 30.8 / 25.8, SkillSynth 33.8 / 29.6 — gains of +3.0 / +3.8 over the strongest baseline come from graph-guided ordering, not skill composition per se [Table 4].
• Unique scenario-skill pair coverage: SkillSynth > single-skill by 31% and > multi-skill by 19%, on matched 1,000-trajectory samples [§4.5].

Why it’s interesting

This is the most explicit existing argument for the recipe Tool-Use Agents and Synthetic Training Data are converging on: the verification gate plus a structured generator that controls trajectory diversity, not raw task count, is what’s load-bearing. It supplies an unusually clean ablation against the same multi-agent harness on three sampling strategies (single-skill, random multi-skill, graph-guided), isolating the contribution of the structural prior — a comparison that Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing (InfTool) and MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling (MiroThinker) don’t run cleanly. It also relates directly to On Data Engineering for Scaling LLM Terminal Capabilities (Nemotron-Terminal) which it explicitly critiques as “LLM-generated taxonomies that diverge from real-world usage”; SkillSynth’s counter-bet is that human-curated skill seeds (ClawHub + GitHub) plus graph-structured composition beat top-down taxonomy synthesis. The Hy3 Preview adoption is a production-scale validation point — same family of method as Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing‘s InfTool used in real training. The error-mode analysis (partial implementation + inline self-test over-trust) is a useful diagnostic that no other filed terminal-agent paper supplies.

See also

• Tool-Use Agents — SkillSynth’s training recipe lives in the parameter-space camp; provides the cleanest synthesis-diversity-ablation in that camp.
• Synthetic Training Data — adds graph-structured trajectory sampling as a new mechanism for the “curation/structure gate dominates raw scale” thesis.
• Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing — InfTool’s three-role pipeline is the closest sibling for tool-use-trajectory synthesis; SkillSynth differs by structurally controlling scenario × skill coverage rather than relying on user-simulator diversity.
• On Data Engineering for Scaling LLM Terminal Capabilities — Nemotron-Terminal is the LLM-taxonomy baseline SkillSynth positions against.
• MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling — MiroThinker scales interaction depth on synthesized multi-hop trajectories; SkillSynth scales scenario × skill coverage on a different task family.
• Training-Free Group Relative Policy Optimization — Training-Free GRPO is the context-space counterpart; SkillSynth is squarely parameter-space SFT.
• VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agents — VisGym’s per-task heuristic solvers play the same “structured trajectory generator” role for visual tool-use that SkillSynth’s skill graph plays for terminal tool-use.
• Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Sampling — Smaller-Weaker-Better uses a weaker synthesizer for compute-optimal sampling; SkillSynth uses MiniMax M2.7 (strong-but-cheap) for the trajectory step and is silent on the synth-strength axis.