Skip to content
Bud's Wiki

UserProxyBench: Evaluating LLM User Simulators for Agent Benchmarks and Training

TL;DR

Section titled “TL;DR”

UserProxyBench is an evaluation layer over the τ-bench family that scores the user simulator — the second LLM that plays the customer role in interactive agent benchmarks — rather than the agent under test. The proposed User Fidelity Score (UFS) uses task-grounded rubric criteria to measure whether the simulated user actually executed its assigned role, scored independently of agent success. Holding the agent fixed at GPT-5.5 and varying only the user proxy across 375 enterprise tasks shifts mean task reward by 15.2 points, and 24.4% of successful episodes contain a user-specification violation — dominantly “premature disclosure” (the user reveals information before the agent asks), which reduces agent tool-call count by 1.06 on average while preserving reward. The paper also identifies a cost–fidelity Pareto frontier across seven proxies, letting practitioners pick the cheapest simulator meeting a fidelity target.

Key claims

Section titled “Key claims”
  • Current interactive agent benchmarks and multi-turn RL training place a second LLM in the user role but score only the agent, leaving unmeasured whether the user proxy correctly executed its role [Abstract, §1].
  • Holding the agent fixed at GPT-5.5 and varying only the user proxy across 375 enterprise τ-bench-family tasks changes mean task reward by 15.2 points, showing user-proxy choice is a first-class confound in agent-benchmark comparisons [Abstract].
  • 24.4% of episodes marked successful by the benchmark’s agent reward contain a user-specification violation under UFS, so agent-side reward alone does not certify a clean evaluation [Abstract].
  • The dominant failure mode is premature disclosure — the user proxy provides information before the agent requests it. This barely moves task reward but reduces agent tool-call count by 1.06 on average across successful episodes, silently changing the interaction being measured while preserving the headline number [Abstract].
  • Across seven evaluated proxies UserProxyBench identifies an empirical cost–fidelity frontier, enabling selection of the cheapest simulator that clears a given fidelity threshold [Abstract].
  • UFS is designed as an independent scorer of user-role adherence — the score does not conflate agent success with simulator quality, and uses task-grounded rubric criteria derived from each benchmark’s private user instructions [Abstract, §1].

Method

Section titled “Method”

UserProxyBench is an evaluation layer wrapping the τ-bench family (τ-bench, τ²-bench, τ-airline, τ-retail, τ²-telecom). Each benchmark task ships with a private user-instruction spec that only the user proxy sees; the User Fidelity Score decomposes each spec into task-grounded rubric criteria (e.g. “did not disclose account number before agent asked”, “requested refund in stated currency”, “escalated when constraint was unsatisfiable”). An independent judge scores each simulated conversation against those criteria, producing a per-episode UFS in [0, 1] alongside the benchmark’s existing agent reward.

The experimental setup holds the agent constant (GPT-5.5) and sweeps seven candidate user-proxy models across 375 enterprise tasks, reporting joint distributions of (agent reward, UFS, per-episode cost, tool-call count) per proxy. The paper’s headline observations — the 15.2-point reward swing from proxy choice, the 24.4% successful-but-violating rate, the -1.06 tool-call effect of premature disclosure — all fall out of that sweep.

Results

Section titled “Results”
  • Agent reward swings 15.2 points across proxies with the agent fixed [Abstract].
  • 24.4% of “successful” episodes contain a UFS-detected user-specification violation [Abstract].
  • Premature disclosure reduces successful-episode tool-call count by 1.06 on average — the agent takes a shorter path to the same reward because the user handed over information early [Abstract].
  • A cost–fidelity Pareto frontier is identified across seven proxies; specific per-proxy numbers are in the paper body [Abstract].

Why it’s interesting

Section titled “Why it’s interesting”

This is the first filed instance in the wiki of a benchmark that scores the user simulator rather than the agent — a slot the RL Environment Platforms cluster has been circling but not formalizing. Prior filings on that page (τ-bench-derived RL platforms like Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing InfTool’s three-role User-Simulator + Assistant + MCP-Server loop; AgentOmnia: Scaling Agentic Models for Full-Scenario Applications AgentOmnia’s 5,018 stateful environments; LLM-as-a-Verifier: A General-Purpose Verification Framework‘s continuous LLM verifier) all assume the user proxy is a fixed, correct oracle — this paper says that assumption silently costs 15.2 points of agent-reward comparability.

The premature-disclosure finding sharpens the Tool-Use Agents “interaction depth” debate directly: MiroThinker argues more tool calls per task is a scaling axis, Training-Free GRPO’s learned experiences reduce required tool calls, and now UserProxyBench shows a user-side mechanism that reduces observed tool calls in a way that looks like agent efficiency but is actually simulator failure. Any interaction-depth comparison across papers using different proxies is now ambiguous.

On the VLM-as-Evaluator page’s rubric-vs-holistic-judge axis, UFS lands squarely on the deterministic-rubric-derived-from-spec side (nearest to behavior-judge — compiling Agent Behavior specs into deterministic + semantic judges‘s compiled BEHAVIOR.md predicates) but applied to the user role rather than the agent trajectory — a symmetric application of the same substrate.

See also

Section titled “See also”