SimpleTool: Parallel Decoding for Real-Time LLM Function Calling

TL;DR

SimpleTool targets end-to-end latency of LLM function calling — not per-token throughput — by exploiting two properties of structured function-call outputs: (a) most tokens are low-entropy structural noise (delimiters, parameter names) and (b) arguments are weakly causally dependent on each other. It introduces 17 special tokens that both compress that low-entropy structure (4–6× output-token reduction) and serve as mode selectors for parallel-decoding heads, one each for function name and up to six positional arguments. The streams share the input prefix’s KV cache and decode concurrently. On Qwen2.5/Qwen3 (0.5B–14B) across five function-calling benchmarks, the recipe yields 3–6× end-to-end speedup with competitive or improved accuracy; an AWQ-quantized 4B reaches 61.2 ms P50 on an RTX 4090 — 16 Hz, above the 10 Hz threshold typically required for embodied / game-AI control.

Key claims

Function-call outputs contain three token classes with very different entropy — structural tokens ({, }, :), key tokens (parameter names copied from the schema), and value tokens (function names + argument values) — and only the last carries real information [§2.1.1].
Replacing low-entropy structure with 17 special tokens (<function>, <arg1>–<arg6>, <|null|>, etc.) compresses output length by 4–6× on samples where the unmodified baseline and SimpleTool both produce correct outputs (mean CR 4.66, P50 5.02 across Qwen 0.5B–14B) [Table 1].
The special tokens double as mode selectors: appending <function> vs <argN> to the same prefix makes the model generate that specific component, so function name and arguments become independent streams sharing one prefill and one prefix KV cache [§2.1.2, Fig. 2].
End-to-end latency under parallel decoding is dominated by the longest single head rather than the sum of all heads: T_e2e ≈ T_prefill + max_i n_i · t_decode [Eq. 2 in §2.1.2].
The extra parallel streams are nearly free because autoregressive decode is memory-bandwidth-bound: 8-head parallel decoding achieves 93.0% average TPOT efficiency across Qwen 0.5B–14B vs. single-stream baseline [§2.1.3, Appendix G].
Speedup on the Transformers backend is 2.8–5.4× on RTX 4090 and 3.0–6.4× on H100, with smaller models gaining the most because they are the most decode-bound; vLLM with prefix caching gives 2.7–4.4× and larger models benefit more as the shared-prefix amortization grows [§3.3, Fig. 5a–b, Table 3].
AWQ 4-bit + SimpleTool on Qwen3-4B reaches 61.2 ms P50 — 16 Hz real-time control on a consumer RTX 4090; Qwen2.5-7B reaches 89.3 ms, both clearing the 10 Hz threshold for responsive control [§3.3, Table 3].
Average overall accuracy improves +2.9% over baselines across five benchmarks; average function accuracy improves +7.1%, attributed to special tokens removing structural ambiguity and shortening outputs so there are fewer tokens that can be wrong [§3.2, Table 2].
On Mobile Actions (released Dec 2025 — after training data was fixed), ST-Qwen2.5-0.5B hits 69.3% zero-shot vs. FunctionGemma 58.0% (+11.3 pp), and after LoRA fine-tuning on the official split reaches 86.2% — beating both FunctionGemma (85.0%) and the 8× larger Qwen3-4B (83.7%) [§3.4, Table 4].
Tail latency is much tighter than FunctionGemma: ST-Qwen-0.5B 74.5 ms P90 vs. FunctionGemma 139.5 ms P90 — critical for real-time deployment where worst-case dominates reliability [§3.4, Table 5].
Training a SimpleTool model requires substantially larger LoRA capacity than standard SFT: rank 1024 reaches 87.0% on BFCL vs. 85.5% at rank 64 (default rank 512), because the model must learn 8 distinct head-conditioned output distributions sharing one input; the embedding layer must also be unfrozen for the new special tokens [§2.2.2, §3.5, Table 6].
Speculative decoding is orthogonal and stacks: applied on top of SimpleTool it gives up to 3× additional forward-pass reduction with 93% token acceptance [Appendix J].

Method

The architecture decouples function calling into independent decoding streams (1 for function name, 6 for arguments by default, plus an optional content head), each initialized by appending its own special token (<function>, <arg1>…<arg6>, <content>) to a shared input prefix. All streams share the prefix KV cache; only the appended head-specific token differs. Because autoregressive decode is memory-bandwidth-bound, batching parallel heads within a single request fills otherwise-idle compute without proportionally increasing memory traffic, so the per-head cost is near-zero [§2.1].

Training is LoRA on the MLP layers (large rank, e.g. 512) with the embedding layer unfrozen so the new special tokens get learned representations. Samples are formatted as parallel sequences sharing the prefix and differing only in the appended special token and target. Per-head loss weighting addresses output-length imbalance. To avoid under-trained heads at rare argument counts (0, 5, 6), the public xLAM corpus is augmented with synthetic balanced-distribution samples generated by Qwen3-235B (+1.0% improvement) [§2.2.3, §3.5].

Results

Across five benchmarks (BFCL Non-Live, BFCL Live, BFCL Exec, Mobile Actions, “Others” = SealTools/OpenFunction/ToolAlpaca) and Qwen2.5/Qwen3 (0.5B–14B), SimpleTool delivers an average +2.9% overall accuracy and +7.1% function accuracy vs. the unmodified base [§3.2, Table 2]. Strongest accuracy gains are on Mobile Actions: +7.3 to +13.3 pp across model sizes, and this benchmark post-dates the training data, so the lift is held-out [§3.2]. Speedup: 2.8–5.4× on RTX 4090, 3.0–6.4× on H100 via the Transformers backend; 2.7–4.4× via vLLM with prefix caching; with AWQ 4-bit, Qwen3-4B reaches 61.2 ms P50 on a 4090, enough for ≥16 Hz real-time control [§3.3, Table 3]. Direct head-to-head against FunctionGemma 270M on Mobile Actions: ST-Qwen-0.5B beats FunctionGemma zero-shot (69.3 vs 58.0) and fine-tuned (86.2 vs 85.0), while running faster at both median (51.0 vs 61.1 ms P50) and tail (74.5 vs 139.5 ms P90) — a 1.8× larger model outperforming a custom edge model in both accuracy and latency [§3.4]. Ablations: LoRA rank matters (85.5 at r=64 → 87.0 at r=1024), and synthetic distribution-balanced data adds ~+1.0% [§3.5, Table 6].

Why it’s interesting

Two threads on the wiki converge here. First, every filed Tool-Use Agents paper to date — InfTool, MiroThinker, Training-Free GRPO, VisGym, K2.5 Agent Swarm — treats the ReAct loop as a batch problem and optimizes accuracy / interaction-depth / cost; none addresses per-call decode latency, which is a hard ceiling for embodied / game-AI deployment (5–30 Hz). SimpleTool is the first filed paper to argue that the function-call itself is the latency-critical artifact and that streaming doesn’t help (a partial JSON call is semantically meaningless until complete) — orthogonal to the GRPO-and-trajectory-synthesis design space the rest of the cluster sits in. Second, it lands as a different lever for LLM Inference Efficiency than the speculative-decoding cluster (Speculative Decoding: Performance or Illusion?, Speculative Speculative Decoding, TorchSpec: Speculative Decoding Training at Scale): SD reduces target-model verification tokens but keeps autoregressive serialization; SimpleTool keeps autoregressive decode but reduces output length (~5×) and serialized stream count (down to max-head) — and the paper explicitly shows the two stack (Appendix J, +3× FPR / 93% acceptance on top of SimpleTool). The compression-as-mode-selection trick is also worth noting independently of speedup: it cleanly reframes the standard SFT objective for function calling as a multi-head conditional generation problem, which is why function accuracy lifts +7.1 pp on top of speedup. Caveat: the recipe is tightly bound to APIs where arguments are well-formed unordered key-value pairs; semantically dependent arguments must be consolidated into one head, which trades parallelism for correctness [§4].