WildToolBench: Why No LLM Scores Above 15% on Real Tool Use

Published benchmarks for LLM tool use show top models scoring 60–80% accuracy. Production deployments show something different — tool calls fail, arguments come back wrong, and agents misinterpret what the user actually wants.

The paper "Benchmarking LLM Tool-Use in the Wild" (arXiv:2604.06185, submitted February 13, 2026) explains why the gap exists: existing benchmarks are too clean. They remove exactly the properties that make real user interactions hard.

The authors built WildToolBench from ToolBench-V2 production logs — real user conversations, not synthetic task descriptions. They evaluated 57 LLMs. No model exceeded 15% accuracy.

Why Existing Benchmarks Overestimate Tool-Use Capability

Most LLM tool-use benchmarks share a common structure:

1. Define a set of tools with clear function signatures
2. Write a task description that maps cleanly to one or more tools
3. Ask the model to call the right tool with the right arguments
4. Score pass/fail on correctness

The problem is in step 2. Task descriptions in controlled benchmarks are written to be unambiguous, single-turn, and directly resolvable from the current message alone. Real users do not write requests that way.

The WildToolBench paper identifies three structural properties that controlled benchmarks systematically remove:

1. Compositional tasks — real requests require chaining multiple tools in non-linear dependency graphs
2. Implicit intent — intent is distributed across prior conversation turns, not present in the current message
3. Instruction transition — conversations mix tool-call requests, clarifications, and casual conversation without explicit signals

Challenge 1: Compositional Tasks

A controlled benchmark task might read: "Search for files named *.log in /var/log." That maps to one tool call with two arguments.

A real user request might read: "Get me a summary of the Slack messages from the platform team this morning, check if any of them are mentioned in our runbook, and open a Jira ticket if there's something new."

That request requires:

1. slack_list_messages(channel="platform-team", date="today") — runs in parallel with:
2. docs_search(query="...") — depends on the output of step 1
3. jira_create_ticket(...) — depends on the outputs of both steps 1 and 2

Effloow Lab reproduced the fan-in DAG structure from this example in Python:

task_steps = [
    {"id": "fetch_slack",   "tool": "slack_list_messages",  "depends_on": []},
    {"id": "search_docs",   "tool": "docs_search",          "depends_on": []},
    {"id": "create_ticket", "tool": "jira_create_ticket",   "depends_on": ["fetch_slack", "search_docs"]},
]
# DAG analysis: max_depth=1, max_width=2 (parallel at level 0, fan-in at level 1)

The output confirmed: max_depth=1, max_width=2, levels: {0: 2, 1: 1}. Two parallel leaf nodes must complete before the fan-in step can run.

Existing benchmarks linearize this — tool calls are presented as sequential, independent steps. A model trained only on sequential benchmarks has no reason to discover that steps 1 and 2 can run concurrently, or that step 3 must wait for both.

Challenge 2: Implicit Intent

In controlled benchmarks, every tool argument is present in the current message. In real conversations, arguments are often introduced in prior turns.

The WildToolBench paper shows this pattern frequently:

Turn 1: "I'm planning a trip to Tokyo."
Turn 2: "What's the weather like there?"     ← "there" = Tokyo from turn 1
Turn 3: "Find me hotels for those dates."    ← "those dates" = unresolvable without asking

Turn 3 references "those dates" — but no dates were mentioned in the conversation. A model must either ask a clarifying question or hallucinate a date. Controlled benchmarks do not include this pattern because it requires a clarifying question that has no single correct answer.

The lab PoC scored this dialogue:

per_turn scores: [0, 1, 3]   (unresolvable references penalized 2×)
mean_score: 1.33
unresolvable references: 1

The 2× penalty on unresolvable references reflects the paper's finding that these require either a hallucinated argument (likely wrong) or an additional clarifying turn (which the benchmark's single-turn evaluation does not allow). Both outcomes register as failures in WildToolBench's scoring.

Challenge 3: Instruction Transition

Real conversations are not single-mode. A user starts with a tool request, interrupts with a clarification, drifts into small talk, then returns to the original task.

Turn 1: "Search for recent papers on RL agents."    → tool_call
Turn 2: "Wait, I meant multi-agent RL specifically." → clarification
Turn 3: "How's your day going?"                      → conversational
Turn 4: "Ok back to it — show me results."           → tool_call
Turn 5: "Can you also search GitHub for repos?"      → tool_call

Lab PoC analysis:

• 5 turns, 3 mode switches
• Switch rate: 75% — mode changes on 3 out of 4 turn boundaries

A model that always responds in tool-call mode will try to call a tool when the user asks "how's your day going?" A model that switches to conversational mode after a clarification may fail to re-enter tool-call mode when the user says "ok back to it."

Controlled benchmarks do not test this because they evaluate single tasks in isolation, not extended conversations with mixed intent.

The 15% Ceiling: What It Means

57 LLMs evaluated. No model above 15%. The paper does not publish a per-model breakdown in the sections Effloow Lab verified. The headline finding is the aggregate ceiling — what it reveals is not that models are uniquely bad, but that the three challenges compound.

A model might handle any one of the three in isolation:

• Compositional tasks: models with strong planning can chain tools
• Implicit intent: models with good memory can track entities across turns
• Instruction transition: models with mode detection can switch behaviors

But real conversations include all three simultaneously. Turn 3 of a conversation might be: implicit reference to an entity from turn 1, requiring a tool call that depends on turn 2's output, arriving as a clarification that interrupts the previous task. That is the wild nature the title refers to.

What This Means for Production Agent Design

If WildToolBench's findings transfer to production (and the paper argues they do — the benchmark is grounded in real logs), several common design assumptions need adjustment:

Assumption: single-turn evaluation captures production performance

It does not. Accuracy on clean, single-turn tool benchmarks predicts very little about multi-turn, implicit-intent conversations. Teams should evaluate agents on session-level transcripts from their actual user base, not synthetic tasks.

Assumption: better models solve the problem

Bigger models improve scores, but none cleared 15% in this study. Benchmark performance is not a reliable proxy for wild performance. Model selection needs to account for the specific challenge type your use case skews toward.

Assumption: compositional tasks require a planner agent

Adding a planning step helps, but the bottleneck is often not planning — it is filling tool arguments from long-range context. A model that plans correctly but cannot resolve "those dates" from a 3-turn conversation still fails the task.

What actually helps (from the paper's controlled experiments):

• Conversation history in context — models with longer effective context windows perform better on implicit intent
• Explicit tool topology in the prompt — telling the model which tools can run in parallel reduces scheduling errors
• Mode detection fine-tuning — models specifically trained to detect instruction transitions outperform general models on challenge 3

A Minimal WildToolBench-Style Evaluation for Your Agent

You do not need the full WildToolBench dataset to get signal on these three challenges. A minimal internal evaluation covers:

# 1. Compositional: Build a 3-step DAG task with one fan-in
#    Score: does the agent call the right tools in the right order?

# 2. Implicit: 3-turn dialogue where turn 3 references turn 1's entity
#    Score: does the agent resolve the reference without hallucinating?

# 3. Transition: 5-turn mix of tool-call, clarification, and casual
#    Score: does the agent re-enter tool-call mode correctly at turn 4?

Running 20–30 examples per challenge in your actual agent environment — with real tools, not mocked ones — will give you a much more reliable signal than any published benchmark score.

Verdict

FAQ

Is WildToolBench publicly available?

The paper (arXiv:2604.06185) describes the benchmark methodology. Public release of the dataset was not confirmed in Effloow Lab's investigation. Check the paper's GitHub link (if any) for current availability.

Does this paper apply to agentic frameworks like LangGraph or CrewAI?

Yes. The challenge categories are not model-specific — they apply to any agent that calls tools from natural language instructions. Multi-agent frameworks add orchestration complexity that can amplify all three challenges.

Which challenge is the hardest?

The paper's controlled experiments suggest implicit intent is the most damaging — unresolvable argument references produce failures that cannot be recovered without a clarifying turn. Instruction transition is hardest for models not specifically trained on mode detection.

How does this relate to existing tool-use benchmarks like ToolBench?

ToolBench-V2 produced the production logs that WildToolBench was built from. The paper's point is that WildToolBench preserves the natural complexity of those logs while existing benchmarks (including ToolBench's evaluation sets) clean that complexity away during dataset construction.

Should developers wait for a model that scores above 15%?

No — the paper does not conclude that 15% is a permanent ceiling. It concludes that existing models are not being trained or evaluated on wild task structures. Models specifically trained on WildToolBench-style data, or fine-tuned for the three challenge types, should score significantly higher.

Effloow Lab inspected arXiv:2604.06185 and ran a Python 3.12.8 stdlib structural PoC on 2026-06-04. No LLMs were evaluated. The 15% ceiling is sourced directly from the paper. Evidence at data/lab-runs/wildtoolbench-llm-tool-use-wild-benchmark-poc-2026.md.