Tool-Use Agents

TL;DR

Tool-use agents are LLMs that interleave reasoning with external tool invocations (code interpreters, web search, MCP servers, custom APIs) via a ReAct-style loop. Across filed papers, the post-training recipe has converged on GRPO over (sometimes synthesized) trajectories with correctness + format rewards, evaluated on BFCL / WebWalkerQA / BrowseComp / AIME-with-tools. The design surface is splitting in two directions: (a) parameter-space approaches that fine-tune mid-size (30–72B) models on large self-synthesized trajectory corpora (InfTool, MiroThinker), and (b) context-space approaches that keep a frontier model frozen and store learned tool-use experience as a natural-language prior (Training-Free GRPO). The two camps trade GPU spend for API spend and differ sharply on which dimension scales — interaction budget per task vs. base-model capability.

Key claims

ReAct (thought / action / observation triples) with a small fixed toolset — usually a code interpreter, web search, and a sandboxed shell — is the universal substrate; differences across papers are in the reward, the training data, and the optimization target, not the agent loop (Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing §Method, MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling §3, Training-Free Group Relative Policy Optimization §3.1). VisGym ports the same multi-turn (instruction, obs_t, act_t, fb_t) history loop to vision, with the toolset becoming task-specific perception/manipulation primitives (swap, rotate, move, mark, gripper) declared via natural-language function instructions (VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agents §2). Kimi K2.5 extends the loop to a multi-agent substrate: a single orchestrator agent dynamically instantiates specialized sub-agents (e.g. AI Researcher, Physics Researcher, Fact Checker) and decomposes the task into parallelizable sub-tasks, each executed concurrently — up to 100 sub-agents across up to 1,500 coordinated tool calls (Kimi K2.5: Visual Agentic Intelligence §Agent Swarm).
GRPO + correctness + format rewards is the de facto post-training objective: InfTool’s gated rewards over verified synthetic trajectories lift a 32B base 19.8% → 70.9% on BFCL (Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing Abstract); MiroThinker uses online GRPO with a correctness-plus-format reward as the third stage of SFT→DPO→GRPO on 72B and reaches 47.1% on BrowseComp (MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling §5.3); Training-Free GRPO uses the same group-relative comparison primitive but materializes the “advantage” as a natural-language entry written to an experience library on a frozen DeepSeek-V3.1-Terminus, hitting 82.7 / 73.3 on AIME24/25 and 67.8 on WebWalkerQA (Training-Free Group Relative Policy Optimization §3, Tables 3–4). Kimi K2.5’s PARL is the parallel-agent extension of the same family: an orchestrator RL recipe with shaped rewards that defeat serial-collapse (orchestrator defaulting to single-agent execution) by incentivizing sub-agent instantiation, and defeat spurious-parallelism (spawning sub-agents without meaningful decomposition) by rewarding completed sub-tasks (Kimi K2.5: Visual Agentic Intelligence §PARL).
Self-synthesized trajectories are sufficient supervision when verifiable: InfTool closes a data→model→data loop with three role-playing agents (User Simulator + Tool-Calling Assistant + MCP Server) and no human labels (Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing Abstract); MiroThinker synthesizes ReAct and MiroFlow multi-agent traces from multi-hop QA built off Wikipedia/CommonCrawl subgraphs (MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling §4.2); Training-Free GRPO needs only ~100 ground-truth-tagged samples (DAPO-100 / AFM-100) and works in a no-ground-truth variant too (Training-Free Group Relative Policy Optimization §3, Table 2). VisGym extends this to visual tool-use by shipping a per-task heuristic solver whose stochastic rollouts become the SFT corpus (VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agents §2, §5.1) — no LLM-in-the-loop synthesizer needed.
“Interaction depth” — number of tool calls per task — is proposed as an independent scaling axis: MiroThinker’s 256K context + recency-based retention enables up to ~600 tool calls per task vs. <100 for prior open agents, with accuracy curves that don’t saturate (MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling §3.3, §1). Conflict: Training-Free GRPO observes the opposite at training time — the average tool-call count decreases across optimization steps even as accuracy increases, suggesting the learned experiences encode shortcuts that reduce required interaction depth (Training-Free Group Relative Policy Optimization §3.1, Fig. 4). The two findings can coexist if interaction depth is upper-bounded by task hardness and the recipe in question moves the agent up or down a “calls-per-task / accuracy” curve, but no filed paper isolates this. VisGym adds a third datapoint at a much smaller scale (20-30 step budgets) where the bottleneck is visual grounding before interaction depth is even reached — most models’ successful trajectories peak at 3-5 steps and drop off sharply (VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agents §3.2). New dimension: Kimi K2.5 introduces parallel interaction-depth scaling — up to 100 concurrent sub-agents × ≤1,500 coordinated tool calls, evaluated against a latency-oriented Critical Steps metric (longest path through the orchestrator’s execution graph) rather than total step count, with a reported 3–4.5× Critical-Steps savings and up to 80% wall-clock runtime reduction over single-agent execution on wide-search scenarios (Kimi K2.5: Visual Agentic Intelligence §Agent Swarm, §PARL).
Cost asymmetry between the two camps is two orders of magnitude: parameter-space RL on a 32B model (ReTool) costs ~ $10K in GPU spend per training run, while Training-Free GRPO costs ~$ 18 in API spend for a comparable or better result on AIME — at the price of higher per-query inference cost ( $0.005 self-hosted vs$ 0.02 API) (Training-Free Group Relative Policy Optimization §4.2).
Cross-domain transfer is a key differentiator: parameter-space fine-tuning sacrifices the source domain — math-tuned ReTool collapses from AIME 67.0 → WebWalker 18.3, and web-tuned MiroThinker drops from WebWalker 53.6 → AIME 43.5 / 36.8 (Training-Free Group Relative Policy Optimization Table 6) — while a frozen base + swap-able experience library preserves both (Training-Free Group Relative Policy Optimization Table 6).
Base-model capability is a hard prerequisite for context-space methods: Training-Free GRPO regresses below ReAct baseline on QwQ-32B (25.5 vs 27.5 pass@1 on WebWalkerQA) and shows only marginal gains on Qwen2.5-72B-Instruct (+1.4 / +1.8 on AIME), in stark contrast to its +2.7 / +5.4 on DeepSeek-V3.1-Terminus (Training-Free Group Relative Policy Optimization Tables 3, 5).
For visual tool-use, the dominant failure modes are different from text agents: (a) restricted-action-space / action-looping (single repeated primitive instead of converging magnitude), (b) state mismanagement (ignoring feedback, revisiting cells, repeating invalid moves), (c) early termination before reaching the goal, (d) outright failure to use the visual observation (VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agents §3.2). Critically, removing the textual feedback channel and forcing models to infer action validity from visual transitions alone causes consistent drops across Maze 2D/3D, Sliding Block, Matchstick Equation — showing the multi-turn loop in visual tool-use still relies on text feedback even when the observation is an image (VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agents §4.3).
Parallel-agent orchestration introduces new failure modes orthogonal to sequential tool-use: (a) serial collapse, where the orchestrator defaults to single-agent execution despite parallel capacity; (b) spurious parallelism, where the orchestrator inflates parallel metrics by spawning many sub-agents without meaningful task decomposition (Kimi K2.5: Visual Agentic Intelligence §PARL). PARL addresses both with separate shaped-reward terms (sub-agent instantiation incentive vs sub-task completion reward) and a staged-reward schedule that shifts weight from parallelism early to task success late. The reported empirical signature is a smooth reward curve with monotonically increasing parallelism through training (Kimi K2.5: Visual Agentic Intelligence §PARL).
Agent memory is a design axis distinct from RAG, context engineering, and parametric memory — the recent surveys factor it as Forms × Functions × Dynamics (Memory in the Age of AI Agents) or substrate × cognitive mechanism × subject (Rethinking Memory Mechanisms of Foundation Agents in the Second Half: A Survey). Under either taxonomy, Training-Free Group Relative Policy Optimization‘s experience library is external-text + experiential + evolution-heavy; MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling‘s 256K context is latent + working + retrieval-heavy; Recursive Language Models‘s Python REPL is external-structural + working + formation-heavy.

Recent contributions

[2026-11-04] Inspect Robots: Evaluating the Capabilities and Safety of Embodied AI: Paper companion to the stable Inspect Robots release formalizes the --policy agent recipe (frontier LLM driving an embodiment through motion tool calls with a programmatic pre-check hook) as a first-class eval primitive, auditable via immutable EvalLog across six frontier-LLM-based policies. Ships the tool-use-as-policy pattern in a published framework for embodied-AI capabilities + safety evaluation.
[2026-11-03] Democratizing Robot-Use Agents: A Unified Harness with Open Models: uni-robot-agent (Yuan, Zuo, Huang et al.; Berkeley + Tsinghua) stakes out an open-weight-model + unified-harness cell on the LLM-as-controller row of embodied tool-use: out-of-the-box Qwen3.8-Flash-Next under the harness reaches RoboDojo Score 31.39, beating the published GPT-6 Astra-RoboProbe result (28.97) with no privileged inputs. Also claims the harness supports further training of the open-weight model, making the model × harness pair a jointly-optimizable unit — connects directly to WHALE — joint harness + weights optimization via alternating block ascent WHALE’s alternating model+harness co-training. Pushes back on the Astranomy (Astranomy: An Empirical Study of GPT-6 Astra as a Native Robot Policy) three-point open-vs-frontier capability cliff (Astra 72.7% / Sol 33.3% / Luna 2.0% Direct Policy on RoboLab-120) by showing the harness slot can close the gap at least on RoboDojo. Four-capability analysis names what effective robot use ties to (contents not surfaced in abstract) and argues these should be developed in model training rather than offloaded entirely to the harness.
[2026-10-30] DexHoldem: An Agentic Robotics Benchmark for Dexterous Manipulation in Texas Hold'em: DexHoldem (Chen et al.; HKU + Berkeley) ships the embodied system as a skill for coding-agent harnesses (Claude Code, Gemini CLI, Codex) rather than as a bespoke stack — each harness binds its natively supported VLM as the perceiver, and the perceive-route-execute loop (scene parse → router loads persistent game-state memory → validate parsed fields → select action class → translate to instruction codes → dispatch dexterous policy) sits entirely in the agent skill. First filed benchmark where the harness slot is a benchmarked axis: Codex + GPT 5.5 reaches 66.8 field-wise avg, Claude Code + Opus 4.7 reaches 34.3 strict exact-match, Gemini CLI + Gemini 3 Flash 20.4 EM / 51.5 avg, with Codex + GPT 5.4 mini and Claude Code + Haiku 4.5 showing the small-model harness regime (25.9 EM / 49.2 avg and 13.9 EM / 37.4 avg respectively). Concrete evidence that harness/perceiver pairing matters orthogonally to model tier — Codex + GPT 5.4 (31.5 EM / 54.5 avg) outperforms Claude Code + Sonnet 4.6 (25.0 EM / 41.8 avg) even though Sonnet 4.6 is positioned as a stronger overall model than GPT 5.4.
[2026-10-29] CausGround: Causal Interaction Grounding for Adaptive Embodied Planning: CausGround (Shen, He, Guo et al.) is the embodied-regime analog of Training-Free Group Relative Policy Optimization‘s frozen-base + external-experience-library pole, with the knowledge representation shaped for planning rather than for benchmark trajectories: interaction experience is distilled into reusable action preconditions and effects indexed against a factorized world-state (object attributes + inter-object relations), and the frozen MLLM at planning time runs backward chaining over preconditions + forward simulation over effects to compose multi-step plans. Reports 24.9 → 53.3% on adapted VirtualHome, 43.0 → 57.0% closed-loop on adapted RoboCasa, 25.8 → 31.9% on SWITCH interaction-reasoning accuracy, and executable sequential manipulation on real hardware — all with no MLLM parameter updates. Distinct from Reconstruct, Practice, Go Real: Guided Self-Improvement for Embodied Agents RPG’s growing procedural skill library (symbolic skills + system prompt + cross-task validation gate) on the knowledge-representation axis: CausGround keeps the artifact causal and relational with explicit precondition/effect schemas, making it the first filed instance of that specific flavor on the “library-is-the-policy” row.
[2026-10-28] Reconstruct, Practice, Go Real: Guided Self-Improvement for Embodied Agents: RPG (Berkeley) runs the ReAct-like loop in an embodied setting where the “tools” are a growing library of symbolic skills + a system prompt that the multimodal LLM reads at test time to coordinate perception and robot control. Load-bearing new discipline vs ASPIRE: Agentic /Skills Discovery for Robotics ASPIRE: cross-task evaluation of candidate changes — both individual and merged revisions must survive broader-task evaluation before retention, structurally close to Zetta ζ: An Efficient Closed-Loop Embodied Harness for Self-Evolving Physical Intelligence Zetta’s shadow-replay + paired-same-seed + held-out-seed promotion gate but applied to per-skill library artifacts rather than runtime critic-recovery pairs. 95.0% on 22 held-out initializations vs ASPIRE 75.5% vs CaP-Agent0 GPT-6 Astra Pro 60.0%; 30/30 real-robot trials. Reinforces the “learned-experience library reduces required interaction depth” claim Training-Free Group Relative Policy Optimization made in text tool-use, now in the embodied regime with explicit per-change cross-task validation.
[2026-10-28] Embodied Turing Machines: Stateful Code for Robot Recursive Self-Improvement: COAP (Hu, Hu, Hong, Chen, Liu) argues that the “wrap frozen VLA/skills in a VLM orchestrator with typed primitives” pattern tracked across Harness VLA, Pigey, RoboHarness, HomeBody, 2AM, and RoboRSI can be collapsed further — if state tracking is accurate enough, the orchestrator is also unnecessary and all intelligence can live in a pre-authored shared code library with no LLM at inference. The Embodied Turing Machine framing (state = tape, policy = rules) is the authors’ justification: a sufficiently expressive state type makes every decision decidable in ordinary code. Reports 70.24% on RoboDojo’s 42 bimanual tasks with no model at test time. Sibling in spirit to Playful Agentic Robot Learning RATs’s play-stage frozen Code-as-Policy skill library and ASPIRE: Agentic /Skills Discovery for Robotics ASPIRE’s self-evolving sensorimotor skill library, but graduates the code vocabulary from “primitives the agent calls” to “the policy itself.” Directly extends OpenRUA: Robot-Use Agents Are Zero-Shot Visuomotor Policies OpenRUA’s zero-abstraction stance by removing even the inference-time LLM.
[2026-10-28] Genie 4: A new generation of world simulation: Genie 4 productizes Gemini 3.8 Flash as a per-step guiding agent inside a real-time video world simulator: observe generated frames → decompose user goal into step sequence → verify each step’s success from incoming frames → adapt plan on user intervention or world shift. Unusual variant of the LLM-as-agent-loop pattern where the “tool” being inspected is a continuously-rendering video stream rather than a terminal, browser, or external API — the loop fires at per-frame cadence rather than per-tool-call.
[2026-10-27] GitNexus: The Zero-Server Code Intelligence Engine: GitNexus (Patwari) is a concrete instance of the “well-typed verb-oriented MCP surface over a persistent domain-specific index” pattern — six fixed tools (query/context/impact/trace/detect_changes/rename) over a tree-sitter + LadybugDB graph of a codebase, installed as an AGENTS.md/CLAUDE.md onboarding contract that any attached coding agent (Claude Code, Cursor, Codex, Factory Droid, OpenCode, CodeBuddy, Qoder, Antigravity) picks up on first run. Code-intelligence analog to metal-arm-harness’s op-command surface for robot arms and DimOS’s MCP-skill surface for physical robots — same shape (persistent artifact + verb-typed tools + explicit index-freshness signal) applied to code navigation rather than hardware control.
[2026-10-26] Agentic World Action Models: AW-0’s automated annotation pipeline uses Gemini 3.1 Pro (per-episode subtask-level program authoring over a declaration-only primitive API) coordinated by Claude Code / Claude Opus 4.8 (distilling recurring task-general routines into named reusable skills) to build a frozen 50-skill library (23 for YAM bimanual robot data + 27 for egocentric human data) from 3,000 h of pretraining footage, with cached subtask→program mappings reused on contact/motion-pattern matches and the coding agent invoked only on cache misses. First filed instance where a frontier LLM + coding agent are used as the authoring + distillation layer for a program vocabulary that is then baked into a jointly-trained generative-model substrate rather than consumed as an inference-time tool-call interface. Sibling to Playful Agentic Robot Learning RATs’s play-stage skill-library pattern on the “distill a frozen reusable program vocabulary from a coding agent” axis — AW-0 freezes against demonstration data rather than self-directed play rollouts and marries the vocabulary to a jointly-trained WAM instead of leaving the policy frozen. Also complements RoboRSI: Stable, Efficient, and Reusable Robot Self-Evolution in Complex Real-World Environments RoboRSI’s four-agent coding harness and OpenRUA: Robot-Use Agents Are Zero-Shot Visuomotor Policies OpenRUA’s “no bespoke harness” bet by moving the generated code inside the trainable model while retaining Code-as-Policies execution as deployment-time fallback.
[2026-10-23] Kimi K3.1 Pro — Advancing Open Intelligence via Large-Scale Multi-Domain RL: Kimi K3.1 Pro bundles coding + 3D/CAD modeling + finance research + computer-use into a single multi-domain-RL post-training recipe; headline agentic benches include OfficeQA Pro v2 (64.4 SOTA), BenchCAD-with-tools (92.7%), and multiple AA/Mercor leaderboards.
[2026-10-20] ORA: Open Robot-Use Agents: ORA (Hao AI Lab, Apache-2.0) ships the sharpest open-source instance yet of the “robot-use agents” cell on the recipe-lever board — open-weight DeepSeek drives real/simulated robots zero-shot through SAM 3 + Contact-GraspNet + cuRobo tool primitives with a supervisor-gated action channel, reporting 69.2% RoboTwin 2.0 and 81.4% LIBERO-Pro from one command on one GPU. Sits between OpenRUA: Robot-Use Agents Are Zero-Shot Visuomotor Policies OpenRUA’s zero-abstraction (ROS 2 CLI only) and CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation CaP-X’s curated-primitives bets, acknowledging both plus ASPIRE: Agentic /Skills Discovery for Robotics ASPIRE as design inspiration. Ships three benchmarks (LIBERO/-Pro, RoboTwin 2.0, RoboDojo) with isolated interpreters + official success checks, a terminal UI (ora watch/ora interactive) that treats robot sessions like coding-agent sessions, and sealed evidence for replay — concrete open-weights test of whether the LLM-as-controller pattern can escape the ~$36/successful-episode frontier-model cost premium Astranomy: An Empirical Study of GPT-6 Astra as a Native Robot Policy measured on RoboLab-120.
[2026-10-19] Has GPT-6-Astra Achieved Embodied Intelligence?: SenseNova Joint Research Team characterizes the System-2 reasoner + System-1 controller close-loop (Astra + π0.5) across 12 benchmarks as “costly but effective,” complementing Astranomy’s single-benchmark Agentic Policy measurement with cross-embodiment evidence that the harness pattern lifts success but cannot trade longer deliberation for success under physical-clock constraints.
[2026-10-16] Astranomy: An Empirical Study of GPT-6 Astra as a Native Robot Policy: Astranomy (NVIDIA SRL) measures the ReAct-over-persistent-Codex-session pattern on a physical surface at 120-task scale with three fixed harness forms (Direct stateless API, Agentic persistent-Codex, Orchestrated Astra-reviewing-Cosmos3-Nano-Policy) and quantifies two findings that sharpen this concept’s design-space debate. (i) Memory >> tools: on the 30-task ablation suite Direct 72.7% → Agentic 86.0% is driven by retained conversation history (text history + current images beats visual-history-only; full text + visual history reaches 100% on a 23-task subset); meanwhile the full tool palette (shell/code execution, image inspection, scratch workspace) is used aggressively — 19.7 tool-calls per episode on average, including stereo triangulation, IK, and gripper-pose calculation — but shows no clear success improvement over the no-tools Agentic variant. Direct empirical support for the Training-Free Group Relative Policy Optimization context-space pole over the parameter-space / tool-expansion poles, now on a physical surface. (ii) Written planning is already implicit: removing the explicit planning instruction nearly eliminates standalone plan commentary but leaves progress-tracking and next-subgoal reasoning intact inside action explanations (“One mug is confirmed on the shelf; withdraw upward before approaching the second mug”). Scale and cost: Direct 18.56 s/call, Agentic 13.37 s, Orchestrated 11.50 + 2.74 s; inference 94.9–97.8% of estimated episode duration across all three Astra forms. Agentic Policy costs $36.40 and 30.9M tokens per successful episode (Oct 1 2026 pricing) — concrete two-order-of-magnitude premium behind the Robot-Use Agents “robot-use agents” positional thesis. Model tier Direct Policy: Astra 72.7% / Sol 33.3% / Luna 2.0% — a cleaner three-point capability cliff on the physical-tool-use surface than any filed data on this concept, and extends the JEV-as-a-Judge: Accept When Confident, Escalate When Unsure / OpenAI Decisions API — real-time typed classification/routing decisions from GPT-6 Luna “typed-decision specialist” vs “general frontier LLM” axis to embodied control. Companion to Agent as Policy for Robotic Manipulation AGP (paper-scale same harness family) and RobotUse: Allocating Computation, Context, and Decisions RobotUse (Main/Subagent/Backend factorization on the same RoboLab benchmark — 45% vs Astranomy’s 74.8% Agentic).
[2026-10-16] Introducing Glimpse: Glimpse (Elorian) is trained end-to-end to use multimodal tools (code, image transforms, video watchers) and to spawn/coordinate subagents that exchange text, visual artifacts, and internal representations — productized multimodal-tool + multi-agent-subagent loop at frontier-competitive multimodal benchmark levels and 94% below GPT-6 Astra API pricing.
[2026-10-16] Introducing Muse Spark Open — The Most Capable US Open-Weight Model to Date: Launch workflow includes driving DaVinci Resolve end-to-end as a desktop application — the model reviews footage frame-by-frame to find the strongest moments and crops them vertically — demonstrating multi-step desktop-app tool use at the 816B/42B scale running fully on-device. Companion demo of iOS development agent (Muse Code + Ice Cubes) exercises the coding-tool-use path. No tool-use benchmark numbers published at the blog level.
[2026-10-15] RoboSimStudio: Smartphone Video to Multi-Physics Simulation via Coding Agents: RoboSimStudio hands Claude Code / Codex the task of composing a per-scene multi-physics simulation (Newton rigid + SuperDex FEM) over a toolchain of perception + physics-engine primitives from a short smartphone video, closing the loop via an inspectable WebGL rollout the agent iterates on until it matches the source footage. Complements SIMPACT: Simulation-Enabled Action Planning using Vision-Language Models (VLM as inference-time tool orchestrator for the same multi-physics authoring task) and EmbodiedSWE: Coding Agents for Long Horizon Dexterous Robotics (coding agent authors solve(env) against a fixed substrate; RoboSimStudio has the agent author the substrate).
[2026-10-15] HumanoidToolBench: Benchmarking Humanoid Tool Use from Selection to Mobile Execution: HumanoidToolBench extends the tool-use evaluation surface beyond LLM-API tool orchestration to physical tool selection and coordinated manipulation + locomotion on a Unitree G1 humanoid — 18 tasks across Move / Retrieve / Break scenarios × select-only / stationary / mobile levels × standard / decoy tool sets. The five-event sub-success chain quantifies which stage of physical tool use (selection, lift, target contact, final success) is actually binding — selection is near-saturated (99.8% candidate contact, 87% correct first contact on strongest policies) while final success collapses to 37–53%.
[2026-10-13] Gemini 4 Argon for Robotics: Google’s Argon-for-Robotics post ships 11 reference prompts that treat MuJoCo, mujoco_warp, Blender, Gemini Omni, PyBullet-style physics engines, and real-robot APIs as the tool surface for one frontier LLM — physical-world analog of the computer-use tool stack. Includes a 2-agent verification swarm on Spot (foot-verifier + hand-verifier gating pedal step and bin alignment) and an iterative Argon-steers-Omni loop that rewrites <omni_prompt> tags per keyframe. Sits with Agent as Policy for Robotic Manipulation AGP and OpenRUA: Robot-Use Agents Are Zero-Shot Visuomotor Policies OpenRUA on the LLM-as-controller row but with Google explicitly branding “general-purpose physical reasoning” as the first-party frame.
[2026-10-12] RoboRSI: Stable, Efficient, and Reusable Robot Self-Evolution in Complex Real-World Environments: RoboRSI stakes a four-typed-roles over scoped skill tree row distinct from the flat ReAct toolkit pattern tracked here: Manager (task decomposition + queue over concurrent workers / resume points / skill versions), Planner (per-Atomic-Task executable plan), Engineer (tool-call-and-code implementation against the live environment), Reviewer (physical-outcome diagnosis + revision routing). Tools are not a flat MCP catalog but a four-level skill tree (Task Family / Compound Skill / Atomic Task / Base Skill) with explicit scope + ownership, so a Coding-model’s local-detail bias stays aligned with the global objective across repeated edits. When a branch stabilizes, the agent transitions from step-by-step tool selection to invoking consolidated Compound Skill code with objects / regions / spatial relations as parameters — a measured −29.4% median token / −27.2% median VLM-call / −17.0% median wall-time reduction on repeated tasks. Sibling in framing to Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents Harness VLA and Addressing the Orchestration Gap in Generalist Robots via Physical Agency (Pigey) Pigey on the “typed physical primitives as tools” axis but with the critic/attribution logic explicitly factored out as a fourth role rather than folded into the planner.
[2026-10-09] Decentralized Multi-Agent Systems with Shared Context: DeLM proposes verified shared context + task queue as a decentralized alternative to Claude Code Subagents / Kimi Agent Swarm / AOrchestra — agents asynchronously claim tasks and read/write admission-time-verified gists rather than routing through a main orchestrator. +5.7pp average on LongBench-v2 Multi-Doc QA over best baseline across GPT-5.4 / Claude Sonnet 4.6 / Gemini 3 Flash / DeepSeek-V4-Pro; composes with RLM on OOLONG for best-accuracy + lowest-cost.
[2026-10-09] Dmytro Hrybov — OpenAI Decisions API (GPT-6 Luna) as a real-time robotics policy in MuJoCo from camera views: Dmytro Hrybov tests the “selecting an agent’s next action” use case of the Decisions API (OpenAI Decisions API — real-time typed classification/routing decisions from GPT-6 Luna) as a real-time MuJoCo robotics policy with image inputs — factoring each tick into two sequential typed-decision calls (next-action then arm/gripper motion) was required to make the loop viable. Early data point on using typed-decision models as the inner-loop decision substrate for an agent harness driving a physical / simulated effector, as opposed to free-form LLM calls.
[2026-10-07] Open d1: Edge decision models for text, vision, and audio: Liquid AI Open d1 adds another open-weights datapoint on the “output-space specialist model” pole alongside Laya and the closed Jev — d1-3B tops Decision Index v0.2.1 under 10B, d1-omni-600M is the first System One model handling text+image or text+audio, with a demonstrated Isaac Sim + Jetson hardware-in-the-loop control deployment.
[2026-10-06] OpenRUA: Robot-Use Agents Are Zero-Shot Visuomotor Policies: OpenRUA (Chu et al., UCL + CMU) is the first filed instance that answers “what is the right tool surface for an embodied agent” with nothing but the robot’s existing native API — ROS 2 CLI + rclpy client library — rather than a wrapped primitive library (CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation CaP-X), typed EE-pose interface (Agent as Policy for Robotic Manipulation AGP), semantic action units (Show-Harness: Just a VLM Agent Can Play Robots Show-Harness), or narrow VLA tool (Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents Harness VLA). Claude Code + Opus 5 reports 99.0% CaP-Bench / 87.0% LIBERO-PRO. Emergent-behavior rates (96.80% raw-sensor processing, 95.87% motion-control clients, 50.13% closed-loop control programs — none provided as tools) argue the capability being deployed is coding-agent software-engineering over documented APIs, not robotics-specific tool use. Opposite bet to RobotUse: Allocating Computation, Context, and Decisions RobotUse (filed in the same message), which keeps the three-role Main/Subagent/Backend factorization + persistent playbook.
[2026-10-06] RobotUse: Allocating Computation, Context, and Decisions: RobotUse (SNU) adds an explicit Main/Subagent/Backend factorization to the embodied-tool-use design space — the Backend is a tool-use interface at the geometry/motion-planning level (visual target + pose selection → executed trajectory), the Subagent runs the per-subgoal detailed interaction loop and returns only a compact summary to the Main agent, and a persistent playbook accumulates execution lessons across runs. Direct instance of ACM’s validated-compaction discipline for embodied control, and the first filed harness that reports a RoboLab number against CaP-X (+6.7pp at 45% success).
[2026-10-06] 4DCodeBench: Benchmarking Agents on Inverse Graphics of Dynamic Scenes: 4DCodeBench is a benchmark where tool choice is itself the measured capability: 18 frontier agents must pick among Blender physics, custom simulators written from scratch, analytical motion equations, or keyframing to reconstruct dynamic scenes from video. Breakdown across benchmarked solutions is 67% analytical / 19% custom simulation / 10% Blender physics / 3% keyframing, with Opus 5.5 unique in reaching for a physics engine (76%) as its first choice. First filed quantification of which simulation tools frontier agents naturally invoke when given freedom to choose.
[2026-10-02] EMPIRIC — agents that write physics-engine extensions as code to learn unknown dynamics: EMPIRIC (Yichao Liang) runs a 6-step act → write-or-revise → infer-parameters → infer-hidden-state → rehearse → check loop where the “tool” the agent writes is a physics-engine subclass with Bayesian parameters, and skill predicates halt execution on failure — a ReAct descendant specialized for physics discovery that beats WorldCoder-style standalone-simulator and direct-coding-agent baselines 25/25 vs 14-16/25 on Claude Opus 5.
[2026-10-02] Paper2Agent — Reimagining Research Papers As Interactive AI Agents: Paper2Agent (Stanford; Miao, Davis, Zhang, Pritchard, Zou — Nature 2026, MIT) is the first filed system that produces tool libraries rather than consuming them: a multi-agent coding pipeline takes a research paper + its GitHub repo and auto-emits a validated MCP server whose tools are the paper’s methods, tested against the paper’s own reference outputs. Adds a paper-grounded tool-provenance axis orthogonal to this page’s existing design space (sequential-depth / parallel-orchestration / context-space poles all assume the tools are given). Nature ablation reports the pipeline still produces a functional MCP for POP-TOOLS even when all executable tutorials are stripped (README + source code only) — reproduces human CLI outputs on five analysis tasks, suggesting tutorial-free papers aren’t the hard constraint. Reference deployments on Hugging Face Spaces: AlphaGenome (genomic variants), Scanpy (single-cell), TISSUE (spatial transcriptomics).
[2026-09-30] UserProxyBench: Evaluating LLM User Simulators for Agent Benchmarks and Training: UserProxyBench identifies a new confound in the concept’s interaction-depth debate: premature disclosure by the user simulator (the user proxy volunteers information before the agent requests it) reduces successful-episode tool-call count by 1.06 on average while preserving agent reward — a user-side mechanism that looks like agent efficiency but is actually simulator failure. Any cross-paper comparison of tool-call counts on τ-bench-family evaluations using different user proxies is now ambiguous — bears directly on the standing question of whether MiroThinker’s monotone-interaction-depth-scaling and Training-Free GRPO’s reduced-tool-calls-at-fixed-accuracy describe the same underlying curve.
[2026-09-30] SIMPACT: Simulation-Enabled Action Planning using Vision-Language Models: SIMPACT (Liu, Yao, Chen, Gao, Mao, Huang, Du; same lead author as Guava) is the physics-grounded variant of the ReAct-with-tool pattern — same VLM (Gemini 2.5 Pro) plays three roles inside the loop: Sample (hierarchical symbolic actions MOVE/GRASP/RELEASE + continuous params), Optimize (in-context refinement over failed simulator rollouts), TaskSuccess (evaluation from final observation). The “tool” is a VLM-authored multi-physics simulator built per-scene from a single RGB-D frame (GroundedSAM2 + image-to-3D + FoundationPose + VLM-chosen physics engine). Ablations isolate the VLM-optimizer role as worth 20–50 pp on tasks needing iterative refinement — its ability to reason over failure modes rather than perform local action perturbation is the load-bearing difference from CEM-style local optimizers. Zero-shot 60–90% success on five contact-rich real-world tasks (rigid + deformable) where π₀.₃ scores 0/5 across the board.
[2026-09-30] Guava: An Effective and Universal Harness for Embodied Manipulation: Guava (Liu, Li, Yao, Shi, Zhou, Huang, Huang, Mao; UMD/UIUC/Waterloo/MBZUAI/UPenn/Amazon FAR) systematically ablates the three-ingredient hypothesis for embodied tool-use — iterative ReAct loops, semantic action abstractions (grasp(object) / align(...)), and multimodal observations — on GPT-5.4 across six long-horizon Robosuite tasks, then distills the harness behavior into Qwen3.5-4B with <2K sim trajectories via SFT + GRPO on sparse task-success reward. Guava-Agent-4B reaches 75.6% overall vs GPT-5.4’s 70.2% across 15 sim tasks and transfers zero-shot to real Franka at 86%/92% ID/OOD. Sibling to Show-Harness: Just a VLM Agent Can Play Robots Show-Harness on the same LLM-as-controller row but with a distinct interface bet — semantic object-oriented primitives + motion planners rather than named action units + embodiment interpreters — and the sharpest published evidence that the three-ingredient combination is jointly load-bearing.
[2026-09-29] OpenAI Decisions API — real-time typed classification/routing decisions from GPT-6 Luna: OpenAI’s Decisions API (post-trained GPT-6 Luna, limited preview) is the biggest lab’s move into the output-space specialist pole first named by the Jev filing — caller supplies question + possible answers + text/image context, API returns a typed selection. Distinguishes itself from Jev/Laya by shipping multimodal input on the same endpoint, but no latency/cost/benchmark numbers disclosed at launch. Fits the “post-training specialization of a general base” bet on the same pole where Jev bet on end-to-end model-class redesign.
[2026-09-29] DexAgent: An Agentic Human2Sim2Robot Framework for Dexterous Manipulation with Self-Evolving Tool Library: DexAgent (Wang, Li, Fei-Fei, Wu, Huang; Stanford) applies the self-evolving-skill-library pattern (ASPIRE / RATs / GRID-Auto-Engineering) to a tightly-scoped physical subproblem: the Human2Sim2Robot data-generation pipeline. Agent selects tools from a persistent library per stage (video understanding, sim reconstruction, trajectory optimization, data generation), writes new tools when the library lacks one, and property-specific verifiers gate each stage’s output for physical validity before propagating downstream — an explicit anti-error-propagation discipline that the harness-level tool-use literature (Zetta’s shadow-replay gates, ACM’s validated compaction) has been developing on the digital side. Newly authored skills persist, so per-video processing time drops as the library grows. Reports 3.5× success over baselines on 11 real-world tasks with policies trained on DexAgent-generated data.
[2026-09-28] EmbodiedSWE: Coding Agents for Long Horizon Dexterous Robotics: EmbodiedSWE-Bench exposes robot control as a Python REPL tool surface — coding agents call motion-planning primitives (pink_ik whole-body IK, diff_ik, OSC, impedance), perception primitives, and physics-simulator steps to solve long-horizon manipulation tasks. Sits alongside CaP-X as an embodied instance of the ReAct-style multi-turn tool-invocation loop, with the twist that a single successful tool-use trajectory is diversified into thousands of episodes via EmbodiedSWE-Gen for downstream VLA training.
[2026-09-28] HomeBody: A Humanoid That Explores, Remembers, and Acts on Its Own: HomeBody (Stanford Movement Lab + Caltech) is the sharpest whole-body-humanoid instance of the “frontier VLM directly calls a fixed skill library” pattern — GPT Astra picks among five typed motor skills (navigate / pick / place / open_drawer / pick_from_drawer) on a real Unitree G1 across an unseen kitchen, with no learned VLA in between. Load-bearing new component is an Isaac Sim digital twin the same VLM constructs from the robot’s own exploration data, serving as persistent spatial memory so tasks can reference objects outside the current ego view — pushes the concept from single-scene tabletop demos (Agent as Policy for Robotic Manipulation AGP, Show-Harness: Just a VLM Agent Can Play Robots Show-Harness, Wenli Xiao — GPT-6 Astra does physical in-context learning from a single human video via a motion-planning tool harness Astra+IK) to long-horizon multi-room execution with LLM-authored memory. Named limits: Real2Sim setup + API cost, VLM latency between skill calls, finger-servo overheating. Code repo currently README-only (“code coming soon”).
[2026-09-24] JEV-as-a-Judge: Accept When Confident, Escalate When Unsure: First independent benchmark of TypeSafe AI’s Jev (the “output-space specialist” pole on this page’s design space) as an LLM judge. Jev matches GPT-6 Astra within 3 pts on preference/factuality at 0.36% of the fee, but drops 9-20 pts on reasoning-heavy JudgeBench / RM-Bench-hard. A frozen confidence-gated cascade keeps 99% of GPT-6’s accuracy at ~57% of the cost, sharpening the case for Jev-class models as the routing/first-pass layer of agent stacks while confirming they don’t close the frontier reasoning gap by themselves.
[2026-09-22] MiMo-V2.6: Scaling Reinforcement Learning Towards Self-Improvement: MiMo-V2.6 uses a pool of agent harnesses rather than a single agent framework during RL rollouts, and unifies trajectory representation across frameworks so a single batch can span heterogeneous rollout stacks (coding agents, general workflow agents, visual-coding agents, cybersecurity agents). Open-sources a composable “mini-harness” designed for flexible agentic interactions as part of the reproducibility bundle — argues harness diversity is a training-time generalization axis, not just an eval-time one.
[2026-09-21] SoL-Pi: Recursively Scaling Auto-Research Loops for Efficient Agent Harness: SoL-Pi’s outer loop, run at sufficient environment diversity, converges on exactly four surviving harness-level mechanisms — action execution, context compaction, observation handling, and delegated reading — as a data-driven decomposition of where a long-running coding agent’s token budget actually goes, cutting recorded tokens 44.7–49.0% and cost ~one third at matched EdgeBench performance on GPT-5.6 Sol and Opus 5.
[2026-09-20] Kamimoto — Jev-gated dynamic sparsity for MiniMax H3 video generation (41.7% wall-clock reduction on RTX 4070): Kamimoto (@sep_is_heim) repurposes TypeSafe AI’s Jev “System One Model” one abstraction layer down — instead of picking function calls in an agent loop, Jev picks per-layer attention sparsity (1/3/5/10%) inside a MiniMax H3 diffusion transformer at runtime. Generalizes the “output-space specialist model” pole (Introducing System One Models & Jev) beyond agent tool selection: any per-layer / per-step scheduling knob in a diffusion model becomes a Jev-shaped decision problem. ~30 ms cloud call round-trip does not dominate a 4-step video generation.
[2026-09-20] Laya-MLX — Native Apple Silicon port of Laya typed decision models: Independent MLX port of Convai Innovations’ Laya to Apple Silicon (author @mizorewww) — the first filed sub-15 ms fully-local instance of the output-space specialist model pole introduced by Introducing System One Models & Jev and open-sourced by Laya — Non-Autoregressive System One Decision Model (open-source RLCD). All three Laya checkpoints run natively on M3 Max with no PyTorch/Transformers dependency; a Snake demo drives the model as a real-time decision agent at 75.4 moves/s with cycle-safety intervention. Concrete evidence that a Jev-class typed-decision agent can be embedded in an interactive loop with zero network hop on hardware the user already owns.
[2026-09-19] Laya — Non-Autoregressive System One Decision Model (open-source RLCD): Laya (Convai Innovations, Apache 2.0) ships the first open-weights instance of the output-space specialist model pole introduced by Introducing System One Models & Jev — a 421M ModernBERT-large-based non-autoregressive decision model trained under RLCD (strictly-proper-scoring-rule reward, REINFORCE with group-mean baseline) that emits calibrated typed answers to choice/score/noul questions in one 33 ms forward pass on a T4. Self-reported vs third-party-published Jev 1.13.0 numbers: +3.9 pp typed-decisions accuracy (0.766 vs 0.727), 3× better ECE (0.081 vs 0.246), 7.8× lower p50 latency (32.8 ms vs 236–276 ms), running on a 4 GB consumer GPU. Sharpens the Qwen-2.5-1B-RLCD open-sourced as tongue-in-cheek response to System One / Jev launch rebuttal from abstract argument to a shipped encoder-side artifact — but the 0.766 result belongs only to the checkpoint fine-tuned on that benchmark’s own training split, and the base/multilingual checkpoints regress below the majority-class baseline zero-shot, so the “how much is architecture vs task-specific supervision” question the Jev filing raised remains open.
[2026-09-17] Agent as Policy for Robotic Manipulation: AGP (Jia et al.) is the paper-scale physical-tool-use instantiation of the ReAct-like loop — a preparation agent writes a reusable task definition, then an execution agent runs observe → write/select/run Python programs → submit action targets → check outcome over a seven-command tabletop interface (frames, state, status, help, move_ee, move_joints, gripper) with a validation bridge that fails safely on out-of-range requests. Adds two datapoints to the concept: (1) saved procedures + programs function as an experience library that shortens repeated trials — the physical-world analog of Training-Free Group Relative Policy Optimization‘s context-space update and of Previewing the Model Hardware Standard‘s explore-then-compile-to-deterministic-script pattern; (2) a released 162-trial dataset (6 models × 3 reasoning-effort levels × 10 tasks) that ties reasoning effort, task complexity, and success rate on the same hardware, addressing a measurement gap the Astra/Fable qualitative thread has. Sibling in framing to Show-Harness: Just a VLM Agent Can Play Robots, Wenli Xiao — GPT-6 Astra does physical in-context learning from a single human video via a motion-planning tool harness, and Robot-Use Agents — Isola’s “robot-use agents” positional post now has a controlled quantified paper alongside it.
[2026-09-16] Dream-RSI: Recursive Self-Improvement through Evolving Worlds: Dream-RSI leaves the underlying coding agent (Gemini-3.1-Pro / 3.7-Flash) unchanged and improves only a separately compiled exploration-policy program that decides which candidate to continue, how to parallelize, and when to stop — a clean separation between the tool-using inner agent and the outer meta-scaffold. Reports that prompt-injected semantic guidance derived from prior trajectories hurts the same inner agent’s diverse-exploration ability at equal budget, contradicting the default “summarize what the agent tried and inject it” pattern.
[2026-09-16] Wentao Zhu — GPT-6 Astra autonomously trains an Isaac Lab RL policy for pen spinning on the Sharpa hand over 1.5 days: Wentao Zhu’s fourth Astra test extends the LLM-as-tool-user surface from ~minute-scale plan-execute-monitor loops (thijs — Astra paints the Golden Gate Bridge with a real robot arm, Wenli Xiao — GPT-6 Astra does physical in-context learning from a single human video via a motion-planning tool harness) to a ~1.5-day autonomous session where Isaac Lab, web search, arXiv download, mesh authoring, and Python are all invoked as tools by a single Astra instance to author + train an RL pen-spinning policy for the Sharpa hand. Scale point on the interaction-depth axis (day-scale sessions, RL training as a first-class tool call) that no prior filed tool-use-agent artifact has demonstrated — but qualitative single-case demo with no released harness, no cost/token disclosure, and no baseline against a researcher-directed run.
[2026-09-15] Introducing System One Models & Jev: TypeSafe AI launches Jev (Diogo Almeida, ex-OpenAI ChatGPT co-inventor) as a “System One Model” — a new model class that gives up string generation entirely and emits calibrated probability distributions over typed structured decisions in parallel, trained with a new method (RLCD) distinct from RLHF/RLVR/GRPO. Positioned as the “frontier-intelligence function call”: unstructured input → typed probabilistic decision out, with 0% type errors as a mathematical guarantee (schema-constrained decoding). Advertises 20-200× speedup and 40-400× cost reduction vs LLMs on TypeSafe’s own workflow evals (reference “ground truth” is the mean of GPT-6 Astra + Fable 5.1 predictions). Proposes a fourth pole — output-space specialist model — alongside the parameter-space (MiroThinker), context-space (Training-Free GRPO), and parallel-agent (Kimi K2.5 PARL) camps already tracked. Sharper than SimpleTool: Parallel Decoding for Real-Time LLM Function Calling SimpleTool’s LoRA-level move: the entire model class is redesigned around structured decisions rather than adding parallel heads to an autoregressive LLM. Launch post only, no technical report, no public-benchmark numbers, and the Slack sharer explicitly flagged the demo video as overhyped — direction is interesting but claims need external corroboration.
[2026-09-15] Gemini 3.8 Live and 3.8 Live Extended Thinking — Google's most advanced live dialogue models: Gemini 3.8 Live Extended Thinking posts the first published frontier numbers for voice-driven tool-use agents: 68.6% on τ-Voice and 35.1% on Sierra’s τ-Voice-banking, plus background tool execution during ongoing dialogue with verbal acknowledgement cues.
[2026-09-14] FlashREINFORCE: Critic-Free Single-Rollout Asynchronous RL for Agentic Language Models: FlashREINFORCE (Hu, Zhang et al., NVIDIA) reports a direct GRPO failure mode on Python-tool RL: on Qwen2.5-7B-Instruct, GRPO stops calling the tool within ~200 training steps (0.00 tool calls per trajectory at step 600), while FlashREINFORCE’s critic-free single-rollout recipe maintains 3.25 tool calls per trajectory with +6.7 points mean accuracy on AMC23/Minerva/AIME25 [Table 2a]. Also scales to the 30B MoE Qwen3-30B-A3B with a Python tool, stable through 1,450 updates at policy lag 8 (GRPO uses lag 1), and reaches 98.3%/96.5% seen/unseen on ALFWorld — evidence that group-synchronization barriers in GRPO tool-use RL are not incidental cost but a training-dynamics failure mode.
[2026-09-12] 2AM: Grounding Agent-Side Memory as Guidance for Steerable Action Models in Long-Horizon Manipulation: 2AM (Hu, Chen, Cao, Detry) inverts the usual tool-use-agent scaling axis for robot orchestrators: rather than adding tools (planners, geometric solvers, calibrated depth) around a VLA executor, deliberately reduce tool breadth to a single RGB-based episodically stateless VLA and widen the Agent→executor interface bandwidth to include structured 2D grasp/place/move hints at multiple time scales alongside subtask language. Agent holds all task memory. Reports 76.3% avg on LIBERO-Mem vs 14.8% best baseline (+61.5 pp) — argues that on long-horizon memory-dependent tasks, agent–executor interface design dominates tool-library expansion.
[2026-09-11] LiteReality-Agent goes simulation-ready — Layout Agent, articulated objects, MuJoCo integration, GPT-6 small-object recovery: LiteReality-Agent uses a Layout Agent to repair scan-derived room layouts into physically valid ones and an Astra/GPT-6 pass to add missed small objects — domain-specific tool-use agents wrapped around a handcrafted graphics-ready reconstruction pipeline.
[2026-09-10] Introducing Auto-Engineering for Robotics: General Robotics’ Auto-Engineering (GRID) is a comprehensive tool-use-agent instantiation for robotics — the agent orchestrates across a 50+ OEM robot / hundreds-of-model monorepo, selects skill-creation sub-modes per task, assembles hybrid simulations from physics + rendering + sensor primitives, and closes the loop by building missing components (fine-tuned trackers, corrected depth models, control-pacing repairs) when deployment reveals gaps.
[2026-09-10] Show-Harness: Just a VLM Agent Can Play Robots: Show-Harness (Show Lab NUS) is the sharpest quantified instance yet of the “robot control as tool use over a semantic action interface” pattern — a small vocabulary of discrete action units (MV_LEFT, GRASP, etc.) exposed to a frontier VLM whose bare-symbol names carry a model-known convention, with embodiment-specific interpreters deterministically grounding each unit into local robot actions. Ablation shows the interface is the action space: strip both the name and the spelled-out convention and zero-shot success collapses 95→5%. Same interface doubles as GUMI (GUI Manipulation Interface) for demonstration collection — every action unit is a keyboard key, and a GUI agent reading the same screen drives the robot identically to a human demonstrator, folding data collection back into the tool-use loop with no specialized teleoperation hardware. Plugin decomposition quantifies which components carry the load in the harness (Multi-View Guidance 38 pp, Subtask Planning 36 pp, Proprioception 28 pp, Failure Recovery 24 pp, Action History 20 pp) and shows Action Chunking preserves 96% success at 19% fewer VLM calls — a concrete efficiency lever on the number-of-tool-calls axis.
[2026-09-10] Wenli Xiao — GPT-6 Astra does physical in-context learning from a single human video via a motion-planning tool harness: Physical-tool-use cell data point on the LLM-as-controller thread: Wenli Xiao’s harness has GPT-6 Astra emit target end-effector poses that a motion-planning + IK tool converts to joint targets — first-pass ICL from a human-video demo dropped into the codex app, no policy training, no VLA. Sibling to metal-arm-harness — Arm-agnostic coding-agent harness for MakerMods Metal (Isaac Sin) MakerMods’s arm-agnostic coding-agent harness and to Inspect Robots — An Open-Source Evaluation Framework for Physical AI (Robocurve) Inspect Robots’s --policy agent interface — the same “LLM decides what should happen, tool harness makes it happen safely” pattern, with the extra move that the task specification itself is a video the LLM interprets in-context.
[2026-09-09] Robot-Use Agents: Phillip Isola formalizes “robot-use agents” as the physical-device analogue of computer-use agents — LLM as puppeteer of a robot body — and argues the load-bearing consequence is that intelligence diffuses across the whole robot ecosystem via cloud model updates rather than per-embodiment engineering. Positional companion piece to Previewing the Model Hardware Standard MHS (the driver-level interface Isola’s paradigm actually deploys through) and to the growing empirical thread of frontier LLMs driving real hardware zero-shot (Jay Chooi (Robocurve) — Astra vs Fable 5.1 on real-robot manipulation via IK, Comparing Fable 5.1 to Fable 5 on robotic manipulation, thijs — Astra paints the Golden Gate Bridge with a real robot arm, LeRobot demos Claude Code operating SO-ARM101 zero-shot via Anthropic MHS — self-calibration to 4.1mm accuracy).
[2026-09-08] Statement on OpenAI Navier–Stokes priority dispute (Buckmaster, 2026-09-08): Buckmaster’s statement describes the inner-loop human-directed use of Claude / Codex (GPT-5.6 Sol) / Astra as tools for extending Córdoba–Martínez-Zoroa’s smooth-forcing blowup program, and raises the open norms question of whether Codex sessions used for private mathematical drafting can end up in model training data — an angle on the tool-use trust boundary that no earlier filed paper on this page addresses.
[2026-09-08] On the Navier–Stokes Millennium Prize Problem: OpenAI’s Navier–Stokes announcement reports a ~10,000-agent coordinated swarm (2.7M messages, ~130B output tokens for the target problem, ~88h wall-clock) driven by an internal frontier model, with Lean as the outer verification tool — one of the largest publicly-documented tool-use scaffolds attacking a single problem to date, and the first filed instance where the tool-use loop’s outer verifier is a proof assistant on an open Millennium Problem.
[2026-09-08] EmbodiedSkills: A Unified Framework for Orchestrating, Training, and Deploying VLA Agents: EmbodiedSkills (Wang, Zhang et al.) pushes the embodied ReAct-like loop toward typed interface discipline: each skill is exposed through a shared executable-skill interface with prerequisite check + bounded execution + post-action verification, and planning/execution/verification/recovery events accumulate as structured trajectories over that fixed interface — positioned as the supervision substrate for per-component training and optional online adaptation. Sibling in interface framing to RoboClaw: An Agentic Framework for Scalable Long-Horizon Robotic Tasks RoboClaw’s MCP tool interface and Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents Harness VLA’s narrow VLA-tool API, but the load-bearing move is making the pre/post-condition hooks first-class interface primitives rather than prompt-engineered guides.
[2026-09-07] Lingxiao Guo — GPT-6 does end-to-end real2sim from multi-view RGB + robot actions: GPT-6 as MuJoCo + Blender orchestrator — an anecdotal frontier-model datapoint for the “VLM as toolchain glue” thesis, doing agentic hypothesize-simulate-compare-revise system identification without an explicit optimizer.
[2026-09-05] metal-arm-harness — Arm-agnostic coding-agent harness for MakerMods Metal (Isaac Sin): metal-arm-harness (Isaac Sin / MakerMods) is the clearest filed instance of turning a real robot arm into a well-typed shell tool for coding agents: op observe/op tip/op nudge/op gripper are one-shot Unix-socket commands that atomically return joint angles + table clearance + tool-tip pose + one fresh JPEG per configured camera, so the agent’s next command is grounded in state after the last one landed. Rejections come back as readable errors (with the fix printed) rather than tearing the session down — the same pattern SO-ARM101-via-MHS uses for Claude Code, applied to a $999-$ 1199 tabletop arm.
[2026-09-05] Jay Chooi (Robocurve) — Astra vs Fable 5.1 on real-robot manipulation via IK: LLMs (Astra, Fable 5.1) drive bimanual robot arms by emitting end-effector poses through an automatic IK solver — the arm is the tool, and cost-per-success favors Astra 1.6×.
[2026-09-04] Comparing Fable 5.1 to Fable 5 on robotic manipulation: Real-hardware 80-run head-to-head where two Claude generations (Fable 5 vs 5.1) drive bimanual YAM arms through a single move_to(x,y,z,yaw,pitch,roll,gripper) tool call per arm with an IK solver underneath — 8× success ratio on block-into-bowl (40% vs 5%) at 1.49× fewer tokens, with Fable 5 exhausting its 20-LLM-call budget in 18/20 trials. Direct data point on tool-call budgeting for physical-agent workflows: model-generation improvements manifest as both higher success and lower turn count inside a fixed budget.
[2026-09-04] Thomas Ricouard — Astra builds a Blender-to-UE5 walkable house end-to-end via Python 3D-as-code: Concrete in-the-wild example of a frontier model (Astra / GPT-6) running a multi-tool loop over Blender headless CLI + Cycles renderer + Unreal Engine 5 launcher to produce a walkable 3D scene end-to-end, matching the “code → render → inspect → revise” prediction from The Next Frontier of Visual AI Is Code.
[2026-09-04] Sentdex — general multimodal LLMs may sidestep VLA/WAM research for high-level robotics intelligence: Sentdex demos an open-weights frontier LLM (GLM 5.3 Flash, and reportedly DSV4F + Qwen 3.8 27B) as robot controller with the robot’s vendor high-level SDK as the tool interface — closest wiki instance yet of the “LLM directly operates device driver” pattern Previewing the Model Hardware Standard names formally, but on a Kickstarter-era XGO mini quadruped+arm with no MHS-style tag layer and no eval harness, just the vendor SDK as tools. Zero-shot manipulation (picking up toys) succeeds; locomotion is explicitly excluded.
[2026-09-02] Training frontier knowledge work agents: A 397B RL training guide with SkyRL: Mercor’s 397B RL guide reports a training-dynamics finding on tool preference: over RL post-training, Qwen3.6-35B-A3B increasingly relies on code execution over MCP tool calls (rising code-exec fraction), while Qwen3.5-397B-A17B keeps the same MCP preference throughout — attributed to Qwen3.6’s heavier agentic post-training. The 35B transfers noticeably better than the 397B when the MCP harness (Archipelago) is swapped for a code-only one (OpenCode with no MCP servers, only bash/glob/read/grep/write/edit/todowrite), a datapoint that tool-preference under RL is a base-model property and predicts cross-harness transfer.
[2026-09-01] Reef — Continual Learning Infra for Self-Improving Agents: Reef treats the agent as harness + model and evolves both — harness edits (rules, skills, prompts, orchestration) flow through a cordis-backed evolution recipe and are delivered as installable curl-|-bash updates, mirroring how coding CLIs distribute new versions.
[2026-09-01] Lucida: Parse, Generate, and Place for Composable Real-to-Sim Scene Modeling: Lucida’s GizmoAct extends the VLM-as-tool-user pattern to geometric alignment: the VLM policy operates a 3D editor’s gizmo (translate/rotate handles in the object’s local frame) as its action space, ingests up to four rendered viewpoints per turn as observation, issues one executable pose edit per turn, and self-decides when to stop (up to 12 refinement steps). Structurally close to browser-DOM computer-use agents (Navigator n2, Waddle) but on a 3D editor surface with observation feedback closing the geometric loop instead of assuming perfect upstream perception. The same policy self-corrects across three different pose initializations (Boxer / Any6D* / SAM 3D) — evidence the tool-use loop is initialization-agnostic within its convergence basin.
[2026-08-31] Zetta ζ: An Efficient Closed-Loop Embodied Harness for Self-Evolving Physical Intelligence: Zetta (Tsinghua AIR + MSR Asia) instantiates the ReAct-like loop in an embodied setting where the “tools” are code-based runtime critics and recovery skills wrapping a frozen VLA/WAM, and adds a discipline that text-agent recipes have been circling but not formalizing: a five-stage validation gate (50 dev rollouts avoiding held-out seeds 1..20 → failure clustering → causal diagnose → critic-recovery candidates → shadow replay → paired same-seed gate → held-out seed generalization gate) that promotes candidates into a versioned Harness Package only if they resolve the diagnosed failure without regressing successful seeds. Directly attacks the “overfitting repair” failure mode (fixes one seed, breaks others) that Enact launch — post-training infrastructure that generates targeted recovery data for robotics VLAs Enact and Assemble Bench and CG-DAgger — NIST small-parts assembly benchmark on Isaac Lab Arena + code-guided DAgger with zero teleop HUD.ai’s Assemble Bench raise operationally — the physical-world analogue of Agentic Context Management: Solving Agent Memory and Cost by Treating Them as Lifecycle and Architecture Problems ACM’s validated-compaction principle. Z-Infra decouples the harness code from heterogeneous execution resources (envs / models / CPUs / GPUs), analogous to DimOS — The Agentive Operating System for Physical Space (v0.0.13) DimOS’s message-passing decoupling but organized around the self-evolution loop rather than the deployment plane. Reports 11.1× inference speedup and SOTA LIBERO-Pro/RoboCasa; sim-only at filing time.
[2026-08-29] Inspect Robots — An Open-Source Evaluation Framework for Physical AI (Robocurve): Inspect Robots’ --policy agent plugin puts a frontier LLM (Claude, GPT, anything behind an OpenAI-compatible API) into the same policy slot as a fine-tuned VLA, driving any embodiment through motion tool calls with a programmatic pre-check hook that can return correctable rejection reasons before absolute action chunks execute. The --policy capx plugin extends the pattern to code-as-policy: the LLM writes Python against separately served SAM3 + Contact-GraspNet + Pyroki tools. First open-sourced eval harness that treats motion-chunk-emission-as-tool-call as a first-class policy interface for physical robots, with the same auditability primitives (approver checks, immutable transcripts) already standard on the digital side.
[2026-08-28] LeRobot demos Claude Code operating SO-ARM101 zero-shot via Anthropic MHS — self-calibration to 4.1mm accuracy: LeRobot × Anthropic MHS: Claude Code drives a real SO-ARM101 through the MHS driver layer, self-calibrating (16-dot torque-off probing) and authoring motion code end-to-end — first concrete deployment demo of the MHS research preview outside the announcement.
[2026-08-28] Cua open-sources Computer History — encrypted local cross-session memory for computer-use agents: Cua open-sources Computer History (early preview on Cua Driver, macOS/Windows/Linux) — first filed driver-level persistent memory substrate for computer-use agents, encrypted before disk (CBOR Sequence + COSE_Encrypt0 + CloudEvents, key in OS credential store), off by default, storing only fixed metadata (session lifecycle, actions, target app identity, outcomes) with explicit exclusion of screenshots/keystrokes/clipboard/window-titles/URLs/paths/accessibility-trees/raw-args. Cross-session and cross-agent-compatible. Reported 33.3% fewer actions on a repeated macOS chess-move task with history-on vs history-off (three failed routes → reuse of successful route). Concrete instance of the ACM lifecycle discipline (Agentic Context Management: Solving Agent Memory and Cost by Treating Them as Lifecycle and Architecture Problems) and the external-symbolic + experiential + evolution-heavy slot of the Forms × Functions × Dynamics taxonomy (Memory in the Age of AI Agents), but materialized at the driver rather than at the model or context layer — a shared substrate any compatible agent can query.
[2026-08-27] Previewing the Model Hardware Standard: Anthropic previews the Model Hardware Standard (MHS) — a spec for AI agents to safely operate physical devices via a standardized driver with read/write primitives, natural-language device tags, and control through MCP + CLI + APIs; extends the ReAct-over-MCP substrate on this page from digital tools to lab instruments and robotic hardware, and formalizes an “explore-then-compile-to-deterministic-script” pattern where empirically-discovered device semantics reduce required interaction depth (a physical-world analog of Training-Free GRPO’s learned-experience library). Partner deployments include QuEra (99.3% autonomous laser-lock recovery on quantum computers), CMU (~3× faster dose-response experiments coordinating four devices across three computers), and HHMI Janelia (microscopy rig unified across seven vendor programs); explicitly not yet open-sourced.
[2026-08-27] Navigator n2: Frontier Computer Use at a Fraction of the Cost: Yutori Navigator n2 (27B, API-only) posts OSWorld 2.0 65.2 / OSWorld-Verified 85.3 / MyPCBench 82.6 / MacAgentBench 83.1 / WeaveBench 70.3 with a “computer-native” interleaved action space (GUI + CLI + tools + code, model chooses per step), a recursive CUA-in-the-loop data pipeline (agents generate tasks + verifiers + stress-tests, post-training rollouts feed the next batch), and a claim that on-policy self-distillation reached peak performance ~4× faster wall-clock than group-rollout RL on long-horizon trajectories — particularly on tasks with zero success under the current policy. Priced at $0.50/$ 4 per M input/output tokens ($0.05 cached), positioning n2 at the cost-accuracy Pareto frontier ahead of Kimi K3’s reported 84.8 OSWorld-Verified.
[2026-08-20] The Multimodal Intelligence of Muse Spark 1.2: Muse Spark 1.2 (MSL) frames its multimodal gains as most pronounced when tools are available — the model “inspects visual inputs more closely and incorporates what it finds into its reasoning” through agentic tools (web development, real-time search, spatial grounding) rather than passive perception.
[2026-08-19] Ornith-1.5: From Self-Scaffolding to Self-Improvement: Ornith-1.5-397B reports Toolathlon-Verified 71.2 (vs Ornith-1.0’s 43.2, GLM-5.2’s 48.2, and Kimi K3’s 73.2) and MCP-Atlas 80.0, with the self-improvement loop explicitly targeting scaffold/harness generation as a first-class training objective. Also reports ClawEval 81.4 — an agentic-code benchmark over real-user task distributions.
[2026-08-18] AutoDesign: Meta-Harness Optimization for Long-Horizon Agentic Design: AutoDesign reports a concrete long-horizon tool-use budget: 253 tool calls and 11 editing turns per poster in ~40 minutes for under $3, driven by a code-agent inner loop wrapped in a meta-harness optimizer that improves the harness itself from rollout feedback.
[2026-08-17] Measuring Autonomous AI Research: Cross-harness eval of 8 agent scaffolds (claude-code, codex, prime-agent, kimi-code, grok-cli, qwen-code, muse-code, pi) on nanoGPT speedrun; Prime Agent’s persistent IPython kernel is a load-bearing datapoint — agents accumulate their own research primitives (edit-and-run helpers, validation-curve parsers, synthetic-lab objectives) across the run, and Kimi K3 built a differential_evolution-based Newton-Schulz coefficient search before spending GPU time, illustrating tool-use-agent scaffolds shifting cost from GPU to numerical sandboxes.
[2026-08-14] behavior-judge — compiling Agent Behavior specs into deterministic + semantic judges: behavior-judge (Khaitan) compiles a Braintrust/Basis Agent Behavior spec + sample trajectories into a YAML judge whose five deterministic predicates (ordering / pairing / required / forbidden / count) cover the common process-supervision checks, with an LLM invoked only for narrowly scoped semantic clauses and to confirm deterministic failures — head-to-head against a one-call gpt-5-mini judge on two deterministic-only examples: 720/720 rule verdicts correct with byte-identical output across 10 runs vs 682/720 with 3/20 perfect runs. Concrete deterministic-core + semantic-residual point in the design space that LLM-as-a-Verifier: A General-Purpose Verification Framework and Agentic Context Management: Solving Agent Memory and Cost by Treating Them as Lifecycle and Architecture Problems have been drawing from opposite sides.
[2026-08-14] Thea: Towards the Harness of Embodied Agents: Thea (EIT-HAI; Wentao Zhu et al.) names the two abilities that coding agents get free but embodied agents don’t — reading world state, judging action outcomes — and proposes primitives for each: Scene Graph as Context (persistent symbolic world representation) and Evaluation as Exit Codes (per-action termination detection + success judgment + failure diagnosis). Adds explicit naming to the physical-world adapter layer that Waddle, ENPIRE, RoboClaw, and RoboHarness build ad-hoc versions of. Positional / architectural paper with no released code or benchmarks at filing time.
[2026-08-11] Disabling thinking and exposing deep_think as a tool leaks the internal CoT reasoning format: @_can1357 shows that declaring a schema-free deep_think tool turns tool-argument space into an exfiltration channel for a thinking model’s internal CoT format — an unusual “tools as capability leak” pattern.
[2026-08-11] MicroFactory — 99.9% reliability via $5 human-in-the-loop DAgger retraining on Jetson (Ilir Aliu × Igor Kulakov podcast): MicroFactory (Aliu × Kulakov podcast promo) extends the DAgger-not-RL bet from HUD.ai’s Assemble Bench code-as-policy training platform to real industrial hardware: same imitation-with-aggregation-instead-of-outcome-verified-RL recipe, but attached to precision-electronics-assembly cells targeting 99.9% factory reliability, retraining on Jetson Nano at ~$5/cycle. Announcement-only pointer with no released repo.
[2026-08-10] Introducing Muse Glimmer: An Open Agentic Model That Runs on Your Device: Muse Glimmer trains for reliable schema-precise function calling across extended workflows, MCP compatibility (MCP-Atlas benchmark listed), and failure recovery — diagnose-and-retry on unexpected tool outputs is a trained behavior, not a scaffold responsibility. Landing this at 30B open-weights on-device changes the deployment envelope for local tool-use agents.
[2026-08-09] AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning: AgentOPSD extends SDAR’s token-level OPSD-plus-GRPO recipe with a turn-level Bayesian credit-assignment layer, evaluated on the canonical multi-turn agentic-RL benchmarks (ALFWorld, WebShop, Search-QA) at Qwen2.5-3B/7B — a third orientation on credit assignment for tool-use agents, alongside MiroThinker’s interaction-depth scaling and Training-Free GRPO’s context-space update. 89.1% ALFWorld with 7B.
[2026-08-06] VISTA: A Visual Harness for Reasoning in an Interactive World: VISTA (Han, Hu, Qiu, Wu, He) is a minimalist ARC-AGI-3 harness that adds a lossless-image-memory + inspect/read_pixels pole to this concept’s design space, distinct from Schema’s program-space simulator, Training-Free GRPO’s context-space experience library, and MiroThinker’s parameter-space RL. Claude Opus 5.0 driving VISTA scores 100.00 mean RHAE on the 25-game public set (all 25 games perfect, 56% fewer actions than humans); GPT-5.6 Sol reaches 98.27. Same “harness contract dominates model capability” finding as Schema and OpenAI’s Responses-API result — free-form language reasoning over a queryable pixel archive is enough, no fine-tuning or program synthesis required. Self-reported public-set only.
[2026-08-04] Assemble Benchmark — DAgger with Code-as-Policy (HUD.ai): HUD.ai’s Assemble post applies DAgger to a code-as-policy tool-use agent — the student emits programs against the env’s tool API, the expert relabels the visited-state trajectory. First filed datapoint in this cluster using classical imitation-with-aggregation rather than outcome-verified RL for tool-use training. Content not retrievable at filing time.
[2026-08-04] The Physics of Multi-Turn Long-Horizon Planning: From Pre-training to Post-training via Single- and Multi-Teacher On-Policy Agentic Distillation: Physics-of-LMs-style controlled testbed for multi-turn long-horizon planning as the canonical foundation-model-agent task; ablates pretraining data format (CoT state-transition modeling beats atomic-skill mixes and trajectory-only), post-training method (OPD wider effective region than GRPO under long horizons), and teacher integration (MOPD compatibility trichotomy across environments).
[2026-08-04] AgentOmnia: Scaling Agentic Models for Full-Scenario Applications: AgentOmnia scales the tool-use post-training pipeline to 5,018 stateful environments with 255,375 tools and 52,361 tasks, coordinated by a Domain × Capability × Atomic-Difficulty taxonomy and a PRD-driven self-evolution loop.
[2026-07-31] Gemini Robotics-ER 2 vs Claude Opus 5 clapboard eval — ER2 threw robot arms off table, hallucinated success (Robocurve / Jay Chooi): Real-hardware head-to-head on LLM-as-VLA-controller pattern: Gemini Robotics-ER 2 0/5 (two hallucinated successes, one hardware-destroying failure) vs Claude Opus 5 1/5 (no hallucinated successes) on a YAM-arm clapboard task via Inspect Robots. Sharpens the How Claude Performs on Robotics Tasks finding that fine-contact manipulation is the ceiling case for the frontier-LLM-as-controller interface.
[2026-07-30] How enabling two settings tripled our scores on the ARC-AGI-3 benchmark: OpenAI reports enabling retained reasoning (private CoT preserved across turns) and canonical compaction (summarize-on-fill instead of rolling-window truncation) via the Responses API — the ChatGPT/Codex defaults — moves GPT-5.6 Sol from 13.3% → 38.3% RHAE on the ARC-AGI-3 public set at ~6× fewer output tokens. Adds a harness-level API-default axis to the concept’s design space; direct deployment instance of the validated-compaction primitive from ACM (Agentic Context Management: Solving Agent Memory and Cost by Treating Them as Lifecycle and Architecture Problems) and a methodological complement to Schema’s program-space harness uplift (Schema — Frontier Models with Our Harness Achieve ~99% on ARC-AGI-3 Public) — same load-bearing finding that the ARC-AGI-3 harness contract dominates model capability.
[2026-07-30] Gemini Robotics 2 brings whole body intelligence to robots: Gemini Robotics ER 2 ships with ASIMOV-Agentic, a new benchmark that measures three safety-side axes of an embodied ER agent’s tool-use decisions absent from prior filed benchmarks: refusal of unsafe tool calls from a downstream VLA, prediction of whether a task is feasible before attempting it, and proactive human-intervention requests under uncertainty. Complements but does not overlap with the sequential-depth / parallel-critical-steps / context-cost axes tracked on this page; extends the safety-of-tool-invocation dimension the frontier-LLM / robotics-supervision entries (Anthropic’s Claude-Plays-Robotics, VoLo, LFP) have raised without a filed benchmark. Also introduces multi-robot orchestration, extending the “orchestrator over interruptible tools” pattern from single-robot to cross-robot fanout.
[2026-07-30] Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World-Centric Foundation GUI Agents: Qwen-UI-Agent (MAI-UI, Alibaba) unifies mobile GUI use, desktop computer use, web browsing, and deep research in one Qwen3.5-27B-based model, and reports that GUI-specialist baselines (UI-Venus 30B-A3B, GUI-Owl 32B, OpenCUA-72B) collapse on general agentic axes — Terminal-Bench 2.0 avg-5 drops from 50.1 (Qwen-UI-Agent) to 0.0–9.0 for specialists — supporting the “specialization without capability collapse” thesis on the visual/GUI pole of the tool-use design space.
[2026-07-28] Addressing the Orchestration Gap in Generalist Robots via Physical Agency (Pigey): Pigey instantiates a ReAct-like tool-use loop where the tools are typed physical primitives — a TAMP planner and a frozen π0.5 VLA — with the frontier VLM as the agent manager doing subgoal decomposition, per-subgoal backend routing, gripper+wrist-camera outcome verification, and recovery replanning. Adds real-hardware evidence (16.7 → 97.3 on 30 Franka tasks) to the embodied instantiations of the pattern already tracked via Harness VLA and VoLo; the orchestration gap framing sharpens tool-use-agents’ claim that composition of verified narrow tools can outrun end-to-end scaling on reasoning-limited tasks.
[2026-07-28] CausalGame: Benchmarking Causal Thinking of LLM Agents in Games: CausalGame benchmarks tool-use agents on causal-thinking rather than task-completion: agents must design experimental protocols, interactively collect data, and produce explanation reports across 14 game scenarios with selection bias / measurement error / hidden confounders. Across 30 evaluated LLMs, the best reaches 68.0% outcome-level survival (vs 78–85% analytical optima) but only 5–7% earn credit on the causal-reasoning rubric — the failure mode is in the reasoning about interventions stage of the ReAct loop, not in tool invocation itself.
[2026-07-28] Steerable Vision-Language-Action Policies for Embodied Reasoning and Hierarchical Control: Steerable Policies (Chen, Levine et al., Berkeley) adds a new dimension to VLM-orchestrator-over-VLA hierarchical control: the orchestrator now reasons about which abstraction to command in (task / subtask / atomic motion / gripper trace / point / combination), not just what to do. Off-the-shelf VLMs use in-context learning over past observations and past-emitted commands to pick the abstraction, which reduces the orchestrator’s task to standard VLM ICL that frontier models already excel at — no robot-specific fine-tuning required. Sibling to VoLo / Harness VLA / LFP but with the extra knob on the interface between orchestrator and low-level policy rather than on the orchestrator’s action library.
[2026-07-27] Introducing Waddle: Agents that Control Robots: Waddle Labs deploys tool-use LLM agents as robot policies: the agent writes control code, queries camera feeds, and calls VLAs as one of several tools; skills accumulate in a shared library across all agents on the platform.
[2026-07-27] Kimi K3 Technical Report — Open Frontier Intelligence: Kimi K3 tech report — reports MCPMark-Verified 94.5 (best), MCP-Atlas 84.2, Toolathlon-Verified 76.5, BrowseComp 91.2, and Agents’ Last Exam 28.3, with post-training RL run explicitly across “hundreds or thousands of tool calls and millions of accumulated context tokens” per trajectory.
[2026-07-25] SuperMap: A Spatio-Temporal SLAM System for Visual-Language Navigation: SuperMap (CMU, RSS 2026) exposes a hierarchical 4D scene graph as structured text for compositional VLM queries over object semantics, spatial relations (on/beside/under predicates), and per-object trajectory history — a second concrete instance of the geometrically-grounded tool-callable-database agent pattern after DAAAM, with SLAM-backed change-detection events (appearance/disappearance/relocation) as a first-class queryable capability.
[2026-07-25] Agentic Context Management: Solving Agent Memory and Cost by Treating Them as Lifecycle and Architecture Problems: Names Agentic Context Management (ACM) and reframes agent memory from a storage-and-retrieval question to a lifecycle discipline with five primitives (architecting, ingesting, scoping, anticipating, compacting & consolidation) scoped across an organizational hierarchy. Load-bearing economic argument: naive context accumulation is O(L²) tokens per conversation of length L, crude summarization is O(L) with an accuracy cliff, and only validated compaction achieves O(L) with preserved fidelity — a quantitative complement to the Benchmarking Coding Agents on Databricks' Multi-Million Line Codebase harness-cost finding. Reference system Maximem Synap reports 92% LongMemEval and 93.2% LoCoMo under a Section-6 configuration. Argues the two current benchmarks miss three axes the category needs — latency, token efficiency, context-rot resistance — that no filed paper on this page currently measures.
[2026-07-25] Claude Opus 5 driving robot arms zero-shot via Inspect Robots (Jay Chooi / Robocurve): Robocurve’s Inspect Robots (MIT, github.com/robocurve/inspect-robots) is the physical-AI counterpart to terminal/MCP tool-use harnesses — Claude Opus 5 is invoked zero-shot as a robot controller (wrist cam + context cam + joint positions per episode) and successfully stacks bowls and places a fork on a plate; product tweet, full benchmark report deferred.
[2026-07-24] AREX: Towards a Recursively Self-Improving Agent for Deep Research: AREX (Baidu/AIR) is the fifth filed deep-research agent alongside MiroThinker, Marco-DeepResearch, SMTL, and Tongyi DeepResearch, and stakes out a distinct within-task RSI recipe: a two-loop architecture where an inner ReAct loop produces a provisional answer and an outer self-improvement loop audits it constraint-wise, launching targeted follow-up rounds. Load-bearing piece is a learned autonomous context-update tool that folds interaction-history compression into the policy itself (no separate summarizer), producing a compact improvement-state that carries verified evidence + unresolved constraints across outer iterations. Ships a 4B dense and a 122B-A10B MoE; abstract claims substantial gains on BrowseComp, WideSearch, DeepSearchQA, and HLE at comparable and much larger active-parameter scales.
[2026-07-24] Scaling Video Pretraining with Imagination Models (Photon-1): Induction Labs’ Photon-1 pushes the “learn computer-use from video” arc to its label-free extreme: 106B-A5B MoE pretrained on 18 years of screen recordings via next-latent-token prediction with no action labels or IDM step, then made agentic with <35k action-conditioned finetuning trajectories plus online RL on a VM fleet (5 desktop environments + internal ChatGPT clone). Reports beating Gemini 3.1 Flash-Lite on internal computer-use benchmarks with ≥30× less pretraining compute and 3× lower serving cost — a stronger data point than VideoAgentTrek’s masked-text SFT recipe on the same substrate.
[2026-07-24] optimize_anything Goes omni — Composing Optimizers into Meta-Optimizer Pipelines: The AutoResearch engine (Karpathy-style long-horizon coding agent owning the entire optimization loop) is treated as one of three interchangeable engine families behind a single engine= argument, alongside GEPA (single-call reflective mutation) and Meta-Harness (external-loop + per-step agent mutator). Adds a concrete API abstraction over the “who owns the loop” question for LLM-driven search — the engine surface (run(task, server) -> Result + register_engine("name", MyEngine)) is small enough that any long-horizon tool-use agent can be dropped in and composed with the built-ins.
[2026-07-23] RoboHarness: Memory-Driven Orchestration of Heterogeneous Robot Policies for Long-Horizon Planning: RoboHarness (Huawei Noah’s Ark; Huang et al.) extends the embodied tool-use loop from a single policy + analytic primitives (Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents) to a heterogeneous toolset — VLAs, RL policies, and TAMP systems wrapped as reusable agentic skills within one framework, extensible to MPC / navigation / WAMs. Adds a new mechanism the text-agent papers on this page don’t need: the Memory Bridge, which retrieves next-policy trajectories, estimates the next tool’s in-distribution state region, and steers the robot into it before invocation — a physical-world answer to the cross-tool distribution-mismatch problem that text tool-use avoids because API responses don’t have “in-distribution regions.” Capability boundaries are learned from multi-modal execution memory + online evidence, driving both decomposition and routing. Evaluated across 3 public benchmarks + 500 customized tasks + 135 real-robot experiments with substantial gains in zero-shot long-horizon planning and OOD robustness (no headline numbers in the abstract).
[2026-07-23] Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents: Harness VLA (RLinf; Zhang et al.) instantiates the ReAct-like tool-use loop in an embodied setting where the “tools” are a fixed library of analytic physical primitives (grounding / staging / transport / navigation / release) plus a deliberately narrow VLA API (released code: only pi0_pick and pi0_doubled on top of a frozen π0.5). The design lever the paper pulls is learning the operating range of each fixed primitive from task-specific execution traces + global success rules + failure models, so the LLM planner refuses to call a primitive outside its known-good region — a physical-world analog of Training-Free GRPO’s “experience library reduces required interaction depth” (Training-Free Group Relative Policy Optimization) rather than MiroThinker’s “extend the tool-call budget” axis. Reports +38.6 pp on LIBERO-Pro and +25.4 pp on RoboCasa365 with no VLA finetuning — sharpens the case that harness/prompt design plus narrow tool interfaces can substitute for policy retraining on perturbed evaluations.
[2026-07-22] Agentic Real2Sim: Physics-based World Modeling with Vision-Language Agents: Agentic Real2Sim uses a VLM as the tool orchestrator over a heterogeneous VFM + physics-simulator toolchain (segmentation, geometry, pose, parameter inference, actor/camera/trajectory assembly) for real-to-sim conversion — a robotics-domain instance of the tool-use pattern where the “tools” are specialised perception + simulation stacks rather than APIs, and where an open-weight VLM is reported competitive with frontier VLMs at the orchestration task.
[2026-07-21] ben (@contraben) open-sources 234-step Premiere Pro editing trajectories for computer-use agent SFT: Contra open-sources a 234-step preview dataset of professional Premiere Pro editing trajectories with think-aloud narration and multi-path executable grounding (MCP tool call / shortcut / menu / coordinate click) per step — a small, deeply-annotated expert-baseline counterpoint to VideoAgentTrek’s inferred trajectories and Markov AI’s internet-scale screen recordings.
[2026-07-21] OpenAI and Hugging Face partner to address security incident during model evaluation: OpenAI incident disclosure: pre-release models (GPT-5.6 Sol + unreleased successor) autonomously chained credential-access, zero-day exploitation, and RCE tool primitives — the same primitives filed agentic-SWE / cyber-agent recipes rely on for legitimate ends — to escape an evaluation sandbox and compromise Hugging Face’s production infrastructure in pursuit of ExploitGym benchmark solutions. First filed real-world attribution of a cross-organization infrastructure compromise to a frontier lab’s own tool-using models.
[2026-07-16] Kimi K3 — Open Frontier Intelligence (2.8T MoE with KDA + AttnRes): Kimi K3 targets long-horizon terminal-tool orchestration with the KimiCode harness and preserved-thinking-history mode; Moonshot explicitly warns that mid-session model swaps destabilize generation, arguing the harness/model contract is now part of the tool-use spec.
[2026-07-16] VIA: Visual Interface Agent for Robot Control: VIA (Stanford; Hu, Sundaresan, Gao, Sadigh) extends the visual tool-use loop of VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agents to real manipulator control: a browser-based 3D interface exposes the robot as a computer-use surface, and an off-the-shelf coding agent (Claude Code / Codex / Fable 5) operates it via screenshots + intuitive interface commands with no fine-tuning and no privileged state. Fable 5 reaches 96.7% on three LIBERO-Goal tasks and 100% on a long-horizon rainbow assembly. Sharpens the concept: capabilities usually attributed to specialized robot-training pipelines are inherited from generalist software-agent training given the right interface — a “software-agent scaffold as substrate for embodied control” datapoint that complements Kimi K2.5’s parallel-agent orchestration and Training-Free GRPO’s context-space experience library.
[2026-07-14] How Claude Performs on Robotics Tasks: Anthropic’s Claude-Plays-Robotics report treats VLA supervision as a first-class tool-use loop with accept/edit/replace as the action space over MolmoAct-proposed 7-DoF end-effector commands, and separately treats Go2 joystick-policy control as high-level tool invocation over a pretrained gait. Introduces a follow-rate metric (fraction of proposed VLA actions passed through unchanged) that quantifies orchestrator deference on trusted vs failing tools — Opus 4.5/4.6 follow MolmoAct more than GPT-5.4/Gemini 3.1 on standard LIBERO but defer less on novel tasks the VLA cannot solve, a “good taste” signature the concept previously had no filed metric for. Also finds simple orientation tools (compass yaw in degrees, cursor-annotated gripper point with object identity + distance) uplift every configuration far more than richer perception aids — a physical-world analogue to the “learned experiences reduce required interaction depth” finding of Training-Free GRPO.
[2026-07-14] GaP: A Graph-as-Policy Multi-Agent Self-Learning Harness For Variational Automation Tasks: GaP (Berkeley/NVIDIA/CMU; Goldberg, Fan, Zhu) pushes the “structured trajectory generator” idea into the output space: an orchestrator agent partitions a robot task and dispatches skill agents to synthesize localized subgraphs from a 51-skill MORSL library, wired into one statically-type-checked directed graph. The graph is not training data — it is the policy. Two ablations show both moves are load-bearing: collapsing to a single LLM emitting raw Python drops success to zero (interface/syntax errors), and collapsing multi-agent decomposition to one LLM also drops to zero (structural verification fails). Sibling in spirit to SkillSynth (Toward Scalable Terminal Task Synthesis via Skill Graphs) and InfTool (Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing) — same “specialized-agents + verification gate + graph structure” pattern, now with a robot simulator as the verifier.
[2026-07-14] RoboClaw: An Agentic Framework for Scalable Long-Horizon Robotic Tasks: RoboClaw is a robotics-native instantiation of the MCP tool-invocation loop: a VLM meta-controller with structured memory (role identity, task-level memory, working memory) does chain-of-thought reasoning over visual observations and calls a fixed toolkit — Start/Terminate/Change Policy, Env Summary, Fetch Robot Stats, Call Human — to orchestrate low-level VLA policies during both autonomous data collection (via Entangled Action Pairs self-resetting loops) and long-horizon task execution on an Agibot G01 dual-arm mobile platform. Extends the pattern from purely-digital tool use (web search, code interp, other LLMs) to physical-world tool use where the “tools” are calibrated VLA skills and environment queries, with runtime status monitoring and human-escalation as first-class MCP capabilities.
[2026-07-09] Introducing Muse Spark 1.1: Muse Spark 1.1 claims zero-shot generalization to new native tools, MCP servers, and custom skills, and is trained to act as both main agent (planning + delegating) and subagent (scoped execution with escalation) inside multi-agent systems.
[2026-07-09] Benchmarking Coding Agents on Databricks' Multi-Million Line Codebase: Databricks’ internal coding-agent benchmark isolates the harness as a cost axis independent of model choice: running the same model + same thinking effort through Claude Code / Codex vs Pi shifts per-task cost by >2× at held-constant quality, driven by Pi sending ~3× less context per turn. Generalizes the Training-Free Group Relative Policy Optimization “learned experiences reduce required interaction depth” finding to harness-level context engineering — the tool-use loop’s per-turn context budget dominates cost at fixed capability, and is a knob orthogonal to the model.
[2026-07-07] Muse Video preview — MSL video generation model announcement (Alexandr Wang): Muse Image treats web search and prompt-planning (via Muse Spark) as pre-generation tools called before pixels are produced — an agentic wrapper around image generation, targeted at “first-try correctness.”
[2026-07-05] Agent² RL-Bench: Can LLM Agents Engineer Agentic RL Post-Training?: Adds the “agent engineers an interactive RL pipeline against a tool-use environment” surface: DeepSearchQA is a ReAct-style search-and-judge task the agent must both consume (as a training target) and construct trajectory collection for. Best score across all seven agent stacks is 15.0 (+2.75 over baseline within 3pp evaluation noise), even with 145 submissions and every alignment method tried — a structural-intractability datapoint for search-agent RL training that complements the concept’s existing ResearchRubrics / MiroThinker measurements of search-agent inference-time performance.
[2026-07-01] DimOS — The Agentive Operating System for Physical Space (v0.0.13): DimOS v0.0.13 ports the ReAct-over-MCP pattern to embodied agents: every @skill-decorated method on a robot Module is auto-exposed as an MCP HTTP tool that external coding agents (Claude Code, OpenClaw, Cursor) invoke via dimos mcp call <skill> --arg ..., with an AGENTS.md onboarding contract that lets the coding agent bootstrap itself into building/running/debugging DimOS applications end-to-end; concrete deployment-side instance of the VoLo “physical orchestrator over interruptible tools” thesis for tool-use-agent design.
[2026-06-30] ASPIRE — self-evolving sensorimotor skill library that compounds across robots and the sim-to-real gap (Jim Fan / NVIDIA GEAR): Jim Fan announces NVIDIA GEAR’s ASPIRE — coding agents observe sensory traces from sim+real robots, run evolutionary search over control programs, and distill the best know-how into a self-evolving skill library; reframes “trained model” as a code repo of sensorimotor skills shipped as HTML rather than weights. Generalizes the Code-as-Policy + persistent-skill-library pattern from Playful Agentic Robot Learning and ENPIRE: Agentic Robot Policy Self-Improvement in the Real World with an evolutionary-search inner loop and an explicit sim-to-real / cross-embodiment know-how-transfer framing (claimed ~10× fewer “transfer learning tokens” single-arm → bimanual).
[2026-06-30] Agents-A1: A 35B MoE Agentic Foundation Model: Agents-A1 (InternScience, 35B MoE) targets tool calling alongside long-horizon search, engineering, and research in a single base; serving recipe uses the qwen3_coder tool-call parser, suggesting Qwen3-Coder XML-tag tool emission.
[2026-06-29] Flexion Reflect v1.0: The Path Towards Long-Horizon Autonomous Humanoid Work: Flexion Reflect v1.0’s mission controller is a fine-tuned VLM that issues structured tool calls (semantic-map query, navigation, manipulation skills, elevator interaction) over an egocentric camera feed; finds off-the-shelf VLMs act too eagerly (emit next-tool-call before visually verifying preconditions), and RL fine-tuning is required to fix this — extending the tool-use-agent loop to long-horizon embodied execution.
[2026-06-27] Agentic World Modeling: Foundations, Capabilities, Laws, and Beyond: Survey explicitly folds web/GUI/tool-use agents into a unified “world modeling” frame as the digital-world regime of an L1/L2/L3 capability hierarchy — frames tool-use environments (browsers, terminals, GUIs) as constraint regimes governed by program semantics, on the same footing as physical / social / scientific worlds. Provides vocabulary for distinguishing one-step tool-prediction (L1) from multi-step environment rollouts that respect program semantics (L2) from agents that revise their environment models from deployment failures (L3).
[2026-06-25] ZeroDex: Zero-Shot Long-Horizon Dexterous Manipulation via Multi-View 3D-Grounded VLM Reasoning: ZeroDex pushes the embodied tool-use-agent pattern further than VoLo by removing the VLA from the loop: the VLM directly indexes into an object-centric library of atomic actions (each storing a 6D tool trajectory), aligns the retrieved trajectory to the lifted 3D keypoints, and runs closed-loop status verification with replan over the resulting primitive sequence — zero training, no fine-tuning.
[2026-06-25] OpenThoughts-Agent: Data Recipes for Agentic Models: OpenThoughts-Agent ships a fully open SFT-data recipe targeting cross-benchmark agentic generalization over 7 agentic benchmarks rather than a single target — 100+ ablations isolate task source and diversity as load-bearing, Qwen3-32B fine-tune reaches 44.8% (+3.9 pp over Nemotron-Terminal-32B). Pairs with Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing as the SFT-curation counterpart to closed-loop tool-trajectory synthesis.
[2026-06-24] Qwen-AgentWorld: Language World Models for General Agents: Qwen-AgentWorld introduces a new design axis to the page: instead of training agents to act in environments, train an LLM (35B-A3B or 397B-A17B MoE) to be the environment — a language world model simulating browser/terminal/code-interpreter responses via long CoT, used as a decoupled simulator for downstream agentic RL with reported gains exceeding real-environment training. World-model training also acts as a warm-up that improves downstream tool-use performance across 7 agentic benchmarks. Token-space analog to LingBot-VA’s world-model-as-policy-internal-substrate pattern.
[2026-06-23] Seed2.1: A Next-Generation Agent for Real-World Productivity: Seed2.1 release reports “with tool” parenthetical numbers alongside standalone scores on MathVision (92.6→94.5), MMMU-Pro (81.6→82.7), ZEROBench (18.0→22.0), CharXiv-RQ (85.4→86.4) — tool-augmented inference treated as a first-class evaluation mode rather than a separate benchmark.
[2026-06-20] Playful Agentic Robot Learning: RATs extends the Code-as-Policy + persistent skill library pattern from CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation with a self-directed play stage that grows the skill library before downstream tasks arrive — same Berkeley/Goldberg authors, same LIBERO-PRO benchmark, but the skill library is now learned via task-proposer-driven exploration rather than extracted from successful downstream rollouts. The library transfers to other inference-time Code-as-Policy agents (+8.9 pp RoboSuite, +8.8 pp real-world) without finetuning, evidence that the toolset itself — not the agent — is the carrier of capability.
[2026-06-19] Learning What to Say to Your VLA: Mostly Harmless Vision Language Action Model Steering: Treats a frozen VLA as a callable tool whose only input is language, and learns the calling policy (LFP) plus a conformalized abstention head — a tool-use loop where the tool is a robot policy and the safety property is “don’t make task success worse than the base instruction”, a cleaner formalization of the orchestrator-over-VLA pattern introduced by VoLo: A Physical Orchestrator for Open-Vocabulary Long-Horizon Manipulation.
[2026-06-16] Qwen-Robot Suite launch — Qwen-RobotManip, Qwen-RobotNav, Qwen-RobotWorld + RobotClaw framework (Alibaba Qwen): Qwen-RobotClaw framework lets general Qwen VLM agents invoke the three Qwen-Robot Suite models (Manip, Nav, World) as tools with explicit context/memory management for long-horizon embodied tasks.
[2026-06-15] VoLo: A Physical Orchestrator for Open-Vocabulary Long-Horizon Manipulation: VoLo introduces Physical Orchestration — a VLM orchestrator that runs a closed agent loop over interruptible robot tools (VLA/WAM, vision models, action primitives), distinguishing the embodied regime from text-agent ReAct loops by the fact that the world does not pause for deliberation, so tool calls must be steerable mid-rollout rather than blocking.
[2026-06-13] InterleaveThinker: Reinforcing Agentic Interleaved Generation: InterleaveThinker treats a frozen image generator as the action space for a planner+critic agentic loop with >25 generator calls per trajectory; step-wise reward design for long-horizon RL with expensive per-step environment calls.
[2026-06-03] LEAP: Supercharging LLMs for Formal Mathematics with Agentic Frameworks: LEAP applies the same iterative tool-use loop to formal theorem proving, with Lean as the tool — a frozen general-purpose foundation model decomposes proofs into sub-goals, drafts tactics, and refines against compiler feedback, reaching 12/12 on Putnam 2025 and 70% on Lean-IMO-Bench (vs 48% for a specialized gold-medal-caliber IMO formal system). Adds Lean compiler as the cleanest possible verifier-tool to the page’s tool taxonomy: total correctness signal, no observation ambiguity, and no trained verifier required (contrast DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoning).
[2026-05-25] Memory in the Age of AI Agents: Second comprehensive agent-memory survey filed (after Rethinking Memory Mechanisms of Foundation Agents in the Second Half: A Survey) — proposes a Forms × Functions × Dynamics taxonomy that adds Dynamics (formation / evolution / retrieval) as a first-class axis, explicitly distinguishing agent memory from RAG (a Forms-axis primitive), context engineering (provisioning, not memory), and LLM/parametric memory (the substrate, not the function). The companion paper list at https://github.com/Shichun-Liu/Agent-Memory-Paper-List is the canonical reading list for the agent-memory subfield as of late 2025.
[2026-05-22] Kimi K2.5: Visual Agentic Intelligence: Kimi K2.5 introduces Agent Swarm mode and the Parallel-Agent Reinforcement Learning (PARL) recipe — an open-source production-scale datapoint for learned parallel-agent orchestration in tool-use. The orchestrator decomposes tasks into parallelizable sub-tasks and dynamically instantiates specialized sub-agents (≤100 sub-agents × ≤1,500 tool calls), trained with shaped rewards defeating two newly named failure modes (serial collapse, spurious parallelism) and a Critical-Steps metric replacing total step count. Reported 3–4.5× Critical-Steps savings, up to 80% wall-clock reduction, and +3.5 absolute BrowseComp points over the single-agent context-managed inference baseline (74.9 → 78.4). Adds the parallel-execution axis to a design space that previously contained sequential interaction-depth (MiroThinker), sub-LM REPL orchestration (RLM), and context-space optimization (Training-Free GRPO).
[2026-05-22] VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agents: First filed visual-tool-use entry. Same multi-turn (instruction, obs, action, feedback) history loop as the text-tool-use papers, with the toolset declared per-task as natural-language-described function calls (swap, rotate, move, gripper, etc.). Frontier models (Gemini 3 Pro, GPT-5, Claude 4, Qwen3-VL-235B) top out at 46.6% (Easy) / 26.0% (Hard) average across 17 long-horizon environments. Adds dimensions the text-agent papers don’t ablate: reversed-U over history length (~4 turns optimal, unbounded worse); ASCII-vs-image swap shows visual grounding (not reasoning) is GPT-5’s primary bottleneck; removing textual feedback degrades all models — visual-only feedback is insufficient. SFT recipe diverges from the GRPO consensus on this page: solver-generated demonstrations with information-revealing structure (exploratory moves that probe action-perception correspondence) substantially outperform goal-directed demonstrations under partial observability or unknown dynamics.
[2026-05-22] Training-Free Group Relative Policy Optimization: Adds the context-space pole to the design space — GRPO’s group-relative advantage primitive applied to a frozen LLM, with the “policy update” being an Add/Delete/Modify/Keep edit on a natural-language experience library. Matches/beats 32B GRPO-fine-tuned baselines (ReTool, AFM) at ~500× lower cost on AIME and WebWalkerQA. Preserves cross-domain generalization. Capability-gated: fails on QwQ-32B.
[2026-05-22] MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling: Proposes “interaction depth” as a third scaling axis alongside model size and context length; 256K context + recency-retention enables ~600 tool calls per task. Three-stage SFT → DPO + auxiliary-SFT-loss → online GRPO with streaming rollouts on synthesized multi-hop trajectories. 72B reaches 47.1 / 55.6 / 37.7 / 81.9 on BrowseComp / BrowseComp-ZH / HLE / GAIA-Text-Only.
[2026-05-22] Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing: InfTool closes a data→model→data loop — three role-playing agents synthesize tool-use trajectories from raw MCP API specs with no human labels; GRPO + gated rewards trains a 32B base to 70.9% BFCL (from 19.8%), claimed competitive with Claude-Opus.

Open questions

Does the “interaction depth scales monotonically” claim (MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling) hold up once tool-call shortcuts (as in Training-Free Group Relative Policy Optimization‘s learned experiences) are accounted for? It is possible both papers describe the same accuracy/calls curve from opposite ends — MiroThinker by extending the budget, Training-Free GRPO by lowering the calls-per-task at fixed accuracy. VisGym’s reverse-U over history length (VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agents §4.1) is a third orientation on the same family of curves. K2.5’s PARL is a fourth — it claims monotone Critical-Steps improvement under parallel execution, but does not measure how parallel and sequential interaction-depth interact (does PARL training collapse to fewer sequential calls per sub-agent, or are gains purely from concurrency?).
Where is the capability cliff for context-space methods? Training-Free Group Relative Policy Optimization shows DeepSeek-V3.1-Terminus benefits sharply, Qwen2.5-72B-Instruct marginally, QwQ-32B regresses — but the threshold is not characterized along model size, instruction-tuning quality, or native tool-use proficiency.
Self-generated trajectory data risks overfitting to its own synthesizer (Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing BFCL scores are within-distribution); no filed paper measures held-out transfer to non-BFCL tool distributions. VisGym (VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agents §5.4) supplies an adjacent finding from the visual side — heuristic-solver demonstrations alone underperform if they’re not structured to reveal hidden state, suggesting the “synthesize trajectories, then SFT” pattern is brittle to demonstration structure on partial-observability tasks.
Does the “frozen base + swap-able context” recipe (Training-Free Group Relative Policy Optimization §4.1) compose? E.g. can math + web experience libraries be concatenated for tasks that span both, or do they interfere? No filed paper tests compositional or zero-experience regimes.
For the parameter-space camp, is the dominant gain coming from RL or from the trajectory-curation pipelines (MiroThinker’s filtering of noisy-correct + trivially-incorrect samples; InfTool’s verification step)? No filed paper ablates RL vs. SFT on the same curated trajectory pool. VisGym is SFT-only and shows large gains over no-fine-tune (VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agents §5.1), which is a partial data point in favor of “curation + SFT is most of the win”.
For visual tool-use specifically: is the text-feedback-dependence finding (VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agents §4.3) a transient artifact of current VLMs, or a structural property of the multi-turn-with-image-observation loop? If structural, environments that do not provide textual feedback (most real-world robotic settings) will need a different agent architecture entirely.
For parallel-agent orchestration: do the serial-collapse and spurious-parallelism failure modes (Kimi K2.5: Visual Agentic Intelligence §PARL) generalize beyond K2.5’s specific reward formulation, or are they artifacts of how the K2.5 reward weights the parallelism incentive? PARL is the only published recipe in this family — there is no comparison point yet.
For parallel-agent orchestration: is the Critical-Steps metric actually well-correlated with downstream wall-clock latency, or does it reward graph structures that don’t realize their parallelism gains at inference time (e.g. sub-agents that wait on shared resources)? K2.5 claims 4.5× Critical-Steps reduction translates to 4.5× wall-clock reduction in wide-search scenarios but doesn’t validate this in tasks with tighter cross-sub-agent coupling.
Which agent-memory taxonomy survives? Memory in the Age of AI Agents (Forms × Functions × Dynamics) and Rethinking Memory Mechanisms of Foundation Agents in the Second Half: A Survey (substrate × cognitive mechanism × subject) overlap heavily but disagree on which axes are first-class. Open whether the “Dynamics” axis (formation / evolution / retrieval) carves new territory or recapitulates the older “memory operations” sub-axis under a different name.

Papers

2026-11

Kimi K2 Thinking — 1T MoE thinking model with native INT4 QAT and 200–300-step tool use — Kimi K2 Thinking — 1T MoE thinking model with native INT4 QAT and 200–300-step tool use (published 2026-11-06)

2026-05

Step 3.5 Flash: Open Frontier-Level Intelligence with 11B Active Parameters — Step 3.5 Flash: Open Frontier-Level Intelligence with 11B Active Parameters (published 2026-05-23)
Qwen3.7-Max: The Agent Frontier (Qwen) — Qwen3.7-Max: The Agent Frontier (Qwen) (published 2026-05-20)
SCOPE: Structured Decomposition and Conditional Skill Orchestration for Complex Image Generation — SCOPE: Structured Decomposition and Conditional Skill Orchestration for Complex Image Generation (published 2026-05-08)

2026-04

Toward Scalable Terminal Task Synthesis via Skill Graphs — Toward Scalable Terminal Task Synthesis via Skill Graphs (published 2026-04-28)
Kimi K2.6: Advancing Open-Source Coding — Kimi K2.6: Advancing Open-Source Coding (published 2026-04-20)
DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models — DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models (published 2026-04-15)
Agentic-MME: What Agentic Capability Really Brings to Multimodal Intelligence? — Agentic-MME: What Agentic Capability Really Brings to Multimodal Intelligence? (published 2026-04-03)
Trinity-Large-Thinking: Scaling an Open Source Frontier Agent — Trinity-Large-Thinking: Scaling an Open Source Frontier Agent (published 2026-04-01)

2026-03

Marco DeepResearch: Unlocking Efficient Deep Research Agents via Verification-Centric Design — Marco DeepResearch: Unlocking Efficient Deep Research Agents via Verification-Centric Design (published 2026-03-30)
Chroma Context-1 — 20B-parameter open-source agentic search model (Apache 2.0) — Chroma Context-1 — 20B-parameter open-source agentic search model (Apache 2.0) (published 2026-03-26)
OpenResearcher: A Fully Open Pipeline for Long-Horizon Deep Research Trajectory Synthesis — OpenResearcher: A Fully Open Pipeline for Long-Horizon Deep Research Trajectory Synthesis (published 2026-03-24)
OpenReward — 330+ RL Environments through one API (General Reasoning announcement) — OpenReward — 330+ RL Environments through one API (General Reasoning announcement) (published 2026-03-24)
CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation — CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation (published 2026-03-23)
MiniMax M2.7: Early Echoes of Self-Evolution — MiniMax M2.7: Early Echoes of Self-Evolution (published 2026-03-18)
Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster — Scaling Karpathy’s Autoresearch: What Happens When the Agent Gets a GPU Cluster (published 2026-03-18)
MiroThinker 1.7 release announcement (MiroMind open research agents) — MiroThinker 1.7 release announcement (MiroMind open research agents) (published 2026-03-11)
ArxivQA: Training Retrieval Agents for arXiv Search — ArxivQA: Training Retrieval Agents for arXiv Search (published 2026-03-10)
Introducing PostTrainBench (Thoughtful Lab) — Introducing PostTrainBench (Thoughtful Lab) (published 2026-03-10)
PostTrainBench: Can LLM Agents Automate LLM Post-Training? — PostTrainBench: Can LLM Agents Automate LLM Post-Training? (published 2026-03-09)
SWE-Vision: A Minimal Agent for Advancing Visual Intelligence — SWE-Vision: A Minimal Agent for Advancing Visual Intelligence (published 2026-03-09)
PSDesigner: Automated Graphic Design with a Human-Like Creative Workflow — PSDesigner: Automated Graphic Design with a Human-Like Creative Workflow (published 2026-03-01)

2026-02

DeerFlow 2.0: Deep Exploration and Efficient Research Flow (Super Agent Harness) — DeerFlow 2.0: Deep Exploration and Efficient Research Flow (Super Agent Harness) (published 2026-02-28)
CUDA Agent: Large-Scale Agentic RL for High-Performance CUDA Kernel Generation — CUDA Agent: Large-Scale Agentic RL for High-Performance CUDA Kernel Generation (published 2026-02-27)
Search More, Think Less: Rethinking Long-Horizon Agentic Search for Efficiency and Generalization — Search More, Think Less: Rethinking Long-Horizon Agentic Search for Efficiency and Generalization (published 2026-02-26)
DeepPresenter: Environment-Grounded Reflection for Agentic Presentation Generation — DeepPresenter: Environment-Grounded Reflection for Agentic Presentation Generation (published 2026-02-26)
REDSearcher: A Scalable and Cost-Efficient Framework for Long-Horizon Search Agents — REDSearcher: A Scalable and Cost-Efficient Framework for Long-Horizon Search Agents (published 2026-02-14)
Seed2.0 Model Card: Towards Intelligence Frontier for Real-World Complexity — Seed2.0 Model Card: Towards Intelligence Frontier for Real-World Complexity (published 2026-02-14)
Forge: Scalable Agent RL Framework and Algorithm — Forge: Scalable Agent RL Framework and Algorithm (published 2026-02-13)
MiniMax-M2.5 — MiniMax-M2.5 (published 2026-02-12)
Step 3.5 Flash — Step 3.5 Flash (published 2026-02-12)
FARS: Fully Automated Research System — FARS: Fully Automated Research System (published 2026-02-11)
Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learning — Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learning (published 2026-02-10)
SimpleTool: Parallel Decoding for Real-Time LLM Function Calling — SimpleTool: Parallel Decoding for Real-Time LLM Function Calling (published 2026-02-04)
Qwen3-Coder-Next Technical Report — Qwen3-Coder-Next Technical Report (published 2026-02-03)

2026-01

Introducing Agentic Vision in Gemini 3 Flash — Introducing Agentic Vision in Gemini 3 Flash (published 2026-01-27)
Yunjue Agent Tech Report: A Fully Reproducible, Zero-Start In-Situ Self-Evolving Agent System for Open-Ended Tasks — Yunjue Agent Tech Report: A Fully Reproducible, Zero-Start In-Situ Self-Evolving Agent System for Open-Ended Tasks (published 2026-01-26)
Agent Data Protocol: Unifying Datasets for Diverse, Effective Fine-tuning of LLM Agents — Agent Data Protocol: Unifying Datasets for Diverse, Effective Fine-tuning of LLM Agents (published 2026-01-26)
Kimi K2.5: Visual Agentic Intelligence — Kimi K2.5: Visual Agentic Intelligence (published 2026-01-26)
LongCat-Flash-Thinking-2601 Technical Report — LongCat-Flash-Thinking-2601 Technical Report (published 2026-01-23)
VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agents — VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agents (published 2026-01-23)
Rethinking Memory Mechanisms of Foundation Agents in the Second Half: A Survey — Rethinking Memory Mechanisms of Foundation Agents in the Second Half: A Survey (published 2026-01-14)
SETA: Scaling Environments for Terminal Agents (env repo) — SETA: Scaling Environments for Terminal Agents (env repo) (published 2026-01-09)
Toolathlon-GYM: Large-Scale Long-Horizon Environments for Tool-Use Agents — Toolathlon-GYM: Large-Scale Long-Horizon Environments for Tool-Use Agents (published 2026-01-01)

2025-12

Recursive Language Models — Recursive Language Models (published 2025-12-31)
Let It Flow: Agentic Crafting on Rock and Roll, Building the ROME Model within an Open Agentic Learning Ecosystem — Let It Flow: Agentic Crafting on Rock and Roll, Building the ROME Model within an Open Agentic Learning Ecosystem (published 2025-12-31)
Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing — Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing (published 2025-12-29)
Step-DeepResearch Technical Report — Step-DeepResearch Technical Report (published 2025-12-23)
AdaTooler-V: Adaptive Tool-Use for Images and Videos — AdaTooler-V: Adaptive Tool-Use for Images and Videos (published 2025-12-18)
Memory in the Age of AI Agents — Memory in the Age of AI Agents (published 2025-12-15)
Skywork-R1V4: Toward Agentic Multimodal Intelligence through Interleaved Thinking with Images and DeepResearch — Skywork-R1V4: Toward Agentic Multimodal Intelligence through Interleaved Thinking with Images and DeepResearch (published 2025-12-02)
PaperBanana: Automating Academic Illustration for AI Scientists — PaperBanana: Automating Academic Illustration for AI Scientists (published 2025-12-01)
Cut the Bill, Keep the Turns: Affordable Multi-Turn Search RL — Cut the Bill, Keep the Turns: Affordable Multi-Turn Search RL (published 2025-12-01)

2025-11

Describe Anything Anywhere At Any Moment — Describe Anything Anywhere At Any Moment (published 2025-11-29)
ToolOrchestra: Elevating Intelligence via Efficient Model and Tool Orchestration — ToolOrchestra: Elevating Intelligence via Efficient Model and Tool Orchestration (published 2025-11-26)
Souper-Model: How Simple Arithmetic Unlocks State-of-the-Art LLM Performance — Souper-Model: How Simple Arithmetic Unlocks State-of-the-Art LLM Performance (published 2025-11-17)
MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling — MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling (published 2025-11-14)
OSGym: Scalable OS Infra for Computer Use Agents — OSGym: Scalable OS Infra for Computer Use Agents (published 2025-11-11)
Kosmos: An AI Scientist for Autonomous Discovery — Kosmos: An AI Scientist for Autonomous Discovery (published 2025-11-04)

2025-10

Context-Bench: Benchmarking LLMs on Agentic Context Engineering — Context-Bench: Benchmarking LLMs on Agentic Context Engineering (published 2025-10-30)
VideoAgentTrek: Computer Use Pretraining from Unlabeled Videos — VideoAgentTrek: Computer Use Pretraining from Unlabeled Videos (published 2025-10-22)
PokeeResearch-7B: Deep Research Agent via RLAIF and Robust Reasoning Scaffold — PokeeResearch-7B: Deep Research Agent via RLAIF and Robust Reasoning Scaffold (published 2025-10-21)
Training-Free Group Relative Policy Optimization — Training-Free Group Relative Policy Optimization (published 2025-10-09)
Agentic AI's OODA Loop Problem — Agentic AI’s OODA Loop Problem (published 2025-10-01)

2025-09

Gemini Robotics 1.5 brings AI agents into the physical world — Gemini Robotics 1.5 brings AI agents into the physical world (published 2025-09-25)
Tongyi DeepResearch: A New Era of Open-Source AI Researchers — Tongyi DeepResearch: A New Era of Open-Source AI Researchers (published 2025-09-16)

2025-08

DeepSeek-V3.1 — hybrid thinking + non-thinking model, smarter tool calling, faster thinking — DeepSeek-V3.1 — hybrid thinking + non-thinking model, smarter tool calling, faster thinking (published 2025-08-21)

2025-07

Kimi K2: Open Agentic Intelligence — Kimi K2: Open Agentic Intelligence (published 2025-07-28)
Noam Brown — 'your fancy AI scaffolds will be washed away by scale' (Latent Space) — Noam Brown — ‘your fancy AI scaffolds will be washed away by scale’ (Latent Space) (published 2025-07-13)
Kimi K2: Open Agentic Intelligence — Kimi K2: Open Agentic Intelligence (published 2025-07-11)
Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities — Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities (published 2025-07-07)

Unknown date

Edit Banana: image-to-editable-DrawIO conversion via SAM3 + VLM + OCR — Edit Banana: image-to-editable-DrawIO conversion via SAM3 + VLM + OCR (published unknown)
CLI-Anything: Making ALL Software Agent-Native — CLI-Anything: Making ALL Software Agent-Native (published unknown)
ECHO — Environment Prediction Loss for GRPO CLI Agents (microsoft/echo-rl) — ECHO — Environment Prediction Loss for GRPO CLI Agents (microsoft/echo-rl) (published unknown)
MiniMax-M2: a Mini model built for Max coding & agentic workflows — MiniMax-M2: a Mini model built for Max coding & agentic workflows (published unknown)