Claude Programmatic Tool Calling: Measured Token Savings

Say you build an assistant that answers a simple business question: "How much did we refund EU customers last week?" To answer, it calls a tool that pulls your order records, sifts through them, and adds up the refunds. The answer is one number. But to get there, the assistant usually drags the entire pile of records (every order, every field) back through the AI model so it can do the filtering and the math in its head.

You pay for that pile. Every record that passes through the model's reading window is billed, whether or not it ends up mattering to the answer. For one small lookup that's a rounding error. For an agent that does this hundreds of times a day across support tickets, billing checks, and CRM lookups, it's a line item.

Anthropic shipped a feature aimed squarely at this waste. It's called programmatic tool calling, and the idea is plain. Claude writes a little code that runs your tools, filters the results, and does the math inside a sandbox, then hands the model only the final answer. The big pile never enters the model's reading window, so you never pay to shuttle it through.

We wanted to know what that's actually worth. So we measured the exact thing the feature removes.

What we built and ran#

We couldn't run Claude's feature end to end in our lab. It needs a specific tool version and a supported Claude model. So we did something more honest and more portable: we isolated the mechanism and measured it on a model API we can call within a strict budget (OpenAI's, model gpt-5.5-2026-04-23).

Here's the setup. We made a fake but realistic pile of data: 150 order records, each with a region, a status, an amount, a customer, and a product code. No real customer data, nothing confidential. The task was the refund question from the top of this article: total up the refunds for EU orders marked "refunded." We worked out the true answer in plain code first: 9 matching orders, $5,016.40.

Then we asked the model the same question two ways.

• The "carry everything" way. We put all 150 records into the model's reading window and asked it to filter and add them up itself. This is what a naive agent does when a tool dumps its whole result back into the conversation.
• The "sandbox did the work" way. We handed the model only the finished result (nine matching orders, $5,016.40) as if a script had already filtered and summed everything out of sight. This mirrors what programmatic tool calling does: the raw pile stays in the sandbox, only the conclusion reaches the model.

Same question. Same correct answer both times. The only thing that changed was how much data we made the model read.

What actually happened#

Both approaches got the answer right: 9 orders, $5,016.40. Accuracy was not the story. Cost was.

In plain terms: reading the whole pile put about sixty times more text in front of the model than reading the conclusion did. The model had to take in 7,462 units of text just to find nine rows that mattered. (Tokens are the unit AI models read and bill in. Think of them as fragments of words.) When the filtering happened out of sight, it read 120. Same answer, roughly 1.6% of the reading (7,462 down to 120).

There was a second, quieter saving. In the carry-everything run the model burned 391 "thinking" tokens grinding through the raw data before it answered. In the sandbox run it spent 10. Handing a model a clean conclusion doesn't just cut what it reads. It cuts how hard it has to work.

Now the honest part, stated up front because it's what makes the number believable. We designed this task so the entire dataset sat in context. That's the mechanism at its most flattering. Real workloads rarely dump one giant blob and nothing else, so real savings run smaller. Anthropic's own published figure is the one to anchor on: across two research benchmarks, adding programmatic tool calling cut input tokens by 24% while slightly improving accuracy. Our 98% is the ceiling. 24% is the everyday. The truth for your workload sits somewhere on that line, depending on how big your tool results are next to everything else in the prompt.

One more thing the raw counts hide, and it cuts against us. In the carry-everything run, 6,912 of those 7,462 input tokens were served from the model's prompt cache, which bills at a fraction of full price. So the dollar gap between the two runs is narrower than the 98% token gap suggests. We're reporting tokens processed, not a bill. Don't convert either number straight into money without applying your provider's cached-versus-fresh rates.

Can this survive your workflow?#

The token math only matters if it maps to something you actually run. A few places where the "let the sandbox filter it" pattern earns its keep:

• Order and refund processing. Pulling a batch of transactions, filtering to the ones that meet a rule, returning a total. That's the exact shape we tested. The bigger the batch, the more the naive approach wastes.
• Support ticket triage. An agent that fetches a customer's full history to answer one question doesn't need the whole history in context; it needs the two relevant events. Filter first, reason second.
• CRM and billing lookups across many records. Anthropic's own example is checking budget compliance across 20 employees: instead of 20 round-trips dragging thousands of line items through the model, one script does the lookups and returns only who went over.
• Internal reporting bots. "How many, how much, which ones" questions over a database export are almost pure filter-then-summarize work. Most of the raw rows never need to reach the model.

Where it does not help: if your agent genuinely needs to reason over every row (summarizing the sentiment of 150 support messages, say, not counting them), the data has to enter the model regardless. Programmatic tool calling saves you when the model's job is to reason over a conclusion, not the raw material. It's a filter-and-aggregate win, not a read-everything win.

If any of the first four sound like something you're building (or planning to), that gap between 98% and 24% is exactly the kind of thing worth measuring against your real tool outputs before you commit to an architecture. That's the sort of proof Effloow's Proof Studio produces: a claim, a bounded run, and a number you can trust. If you'd rather talk it through first, tell us about your workflow.

What to do differently after reading this#

Three practical takeaways:

1. Audit where your tool results are big. The cost is proportional to how much data your tools hand back. A tool that returns 5 fields is fine; a tool that returns a 200-row export on every call is quietly expensive. Find those.
2. Move filtering and math out of the model's head. Whether through Claude's programmatic tool calling, your own pre-processing, or a tool that accepts a filter argument, the goal is the same: the model should read conclusions, not raw dumps.
3. Measure before you believe a percentage, ours included. Our 98% is real but favorable by design. Your number depends on your data. The method in this article is cheap to copy; run it on your own tool outputs.

When to use programmatic tool calling, and when to skip#

Use it when:

• Your agent calls tools that return large intermediate results it then filters or aggregates.
• You run the same tool many times in a loop (per-employee, per-order, per-repo) and only the summary matters.
• Token cost or latency on multi-tool workflows is a real budget line, not a hypothetical.

Skip it (for now) when:

• The model genuinely needs every row to do its job.
• You're on Amazon Bedrock or Google Cloud. As of this writing it isn't available there (Claude API, Claude Platform on AWS, and Microsoft Foundry's Hosted-on-Anthropic deployment only).
• You need Zero Data Retention. The feature isn't eligible for ZDR, so data follows the standard retention policy instead.

Limitations of our test, in plain words#

We're stating these because they're what separate a real measurement from a marketing number.

• We measured the mechanism (big result in context vs. small result in context), not Claude's actual programmatic tool calling feature. The saving is real, but it's provider- and model-neutral, measured on OpenAI's API, not on Claude.
• One task, deliberately shaped so the whole dataset sat in context. That maximizes the gap. Expect less on mixed workloads.
• Token counts and prices differ across models and providers. Don't turn our token numbers into a dollar figure without applying current per-million-token rates.
• No production system was migrated, and no customer pipeline was tested. This is a bounded lab check, and the full run notes and raw usage counters are published in our public lab note.

What Effloow added#

The primary sources (Anthropic's docs and engineering posts) tell you programmatic tool calling reduces tokens and give one benchmark figure: 24% fewer input tokens. What they don't give you is a hands-on feel for why and how much on a task you can picture. Our original contribution is a reproducible, side-by-side worked example with a concrete input (150 order records), a concrete correct output ($5,016.40), and the real API usage counters for both approaches, plus the honest framing that our 98% is a ceiling and 24% is the realistic middle. That contrast is the thing a buyer actually needs to reason about, and it isn't in any single vendor page.

For related reading, see our measurement of how web-search agents waste tokens carrying consumed results and our broader guide to token optimization for production LLM costs.

For your engineers#

Everything below is the reproducible method and the exact configuration. It's separated from the business narrative on purpose; nothing here changes the conclusions above.

The measurement. We used scripts/openai-lab-run.py (OpenAI Responses API, model gpt-5.5-2026-04-23) to send two prompts with an identical task and compare the usage.input_tokens and usage.total_tokens the API returned.

• Naive prompt: task instruction + full 150-record JSON array. Result: input_tokens=7462, total_tokens=7871, reasoning_tokens=391. Request id req_289602e69f1c49b6b5a7c9eb968e29df.
• Programmatic-analog prompt: task instruction + only {"filtered_count":9,"total_usd":5016.4}. Result: input_tokens=120, total_tokens=148, reasoning_tokens=10. Request id req_68f375190756441bb66d215c45229c07.
• Both returned count=9, total_usd=5016.4. Dataset and ground truth were generated deterministically in Python; prompt SHA-256 prefixes and full usage JSON are in the lab note.

Claude programmatic tool calling, verified config (from the Claude programmatic-tool-calling docs, fetched 2026-07-13; not executed in this lab):

{
  "model": "claude-opus-4-8",
  "tools": [
    { "type": "code_execution_20260120", "name": "code_execution" },
    {
      "name": "get_orders",
      "description": "Fetch order-line records",
      "input_schema": { "type": "object", "properties": {} },
      "allowed_callers": ["code_execution_20260120"]
    }
  ]
}

• Add code_execution to tools, and add allowed_callers: ["code_execution_20260120"] to each tool Claude may call from sandbox code. The docs advise choosing ["direct"] or ["code_execution_20260120"] per tool rather than both, for clearer guidance to the model.
• code_execution_20260120 and code_execution_20260521 are interchangeable inside allowed_callers; response blocks tag the caller as code_execution_20260120 regardless of which version you declared.
• Requires code_execution_20260120 or later. Supported models per the docs: Claude Fable 5, Mythos 5, Opus 4.8, Opus 4.7, Opus 4.6, Opus 4.5 (claude-opus-4-5-20251101), Sonnet 4.6, Sonnet 4.5 (claude-sonnet-4-5-20250929).
• Available on the Claude API, Claude Platform on AWS, and Microsoft Foundry (Hosted-on-Anthropic deployment). Not on Amazon Bedrock or Google Cloud at time of writing. Not eligible for ZDR.
• Anthropic's benchmark: on BrowseComp and DeepSearchQA, PTC on top of basic search tools improved performance ~11% while using 24% fewer input tokens.

Reproduce: generate the 150-record dataset and two prompt files, then run scripts/openai-lab-run.py --slug claude-programmatic-tool-calling-token-proof-2026 --prompt-file <file> --max-output-tokens 500 --output data/lab-runs/<slug>.<variant>.openai.json once per prompt, and diff the usage blocks. Full commands and the raw artifacts are in the public lab note.

Sources:

• Programmatic tool calling (Claude Platform Docs)
• Code execution tool (Claude Platform Docs)
• Introducing advanced tool use on the Claude Developer Platform (Anthropic Engineering)
• Improved web search with dynamic filtering (Anthropic)