Claude Programmatic Tool Calling Token Proof 2026

Date: 2026-07-13
Track: api-backed-poc
Purpose: measure, on a real model API, how many tokens a large intermediate tool result costs when it round-trips through the model context, versus when only the final aggregated result reaches context. This is the exact cost that Claude's programmatic tool calling is designed to remove.
Runtime for the measurement: OpenAI Responses API, model gpt-5.5-2026-04-23, via scripts/openai-lab-run.py.

Why measure this on OpenAI and not on Claude's PTC directly

Claude's programmatic tool calling (code_execution_20260120) keeps intermediate tool outputs inside the code-execution sandbox instead of passing them back through the model context. We did not run Claude's PTC feature here (it requires the code-execution tool version and a supported Claude model). Instead we isolated and measured the underlying mechanic — the token cost of putting a large intermediate payload into a model's context window — using a neutral reasoning-model API we can call within budget. The measured numbers below therefore quantify the cost PTC targets; they are not a benchmark of Claude's PTC implementation. Anthropic's own published figure for PTC is a separate, cited data point (see below).

Setup

Synthetic, deterministic scenario (no secrets, no customer data). An agent "fetched" 150 order-line records. Task: sum amount_usd for records where region == "EU" and status == "refunded". Ground truth (computed in plain Python): count=9, total_usd=5016.40.

Two prompts, identical task, differ only in how much of the intermediate data reaches the model:

Naive pattern — the full 150-record JSON payload is placed in the model context; the model must filter and aggregate. Prompt file: naive_prompt.txt (SHA-256 prefix 8d230255a4689273).
Programmatic pattern — an in-sandbox script already filtered and aggregated; only the compact result {"filtered_count":9,"total_usd":5016.4} reaches the model context. Prompt file: programmatic_prompt.txt (SHA-256 prefix 44f7c9b84b8027b6).

Command shape (run twice, once per prompt file):

python3 scripts/openai-lab-run.py --slug claude-programmatic-tool-calling-token-proof-2026 \
  --prompt-file /tmp/ptc-lab/naive_prompt.txt --max-output-tokens 500 \
  --output data/lab-runs/<slug>.naive.openai.json

Artifacts saved: *.naive.openai.json and *.programmatic.openai.json (each stores usage, request id, prompt SHA-256).

Results (real API usage counters)

Metric	Naive (full data in context)	Programmatic (compact result only)	Change
Input tokens	7,462	120	−98.4% (62.2× fewer)
Output tokens	409	28	—
of which reasoning tokens	391	10	—
Total tokens	7,871	148	−98.1% (53.2× fewer)
Answer	`count=9, total_usd=5016.40`	`count=9, total_usd=5016.4`	both correct

Naive request id: req_289602e69f1c49b6b5a7c9eb968e29df
Programmatic request id: req_68f375190756441bb66d215c45229c07
Both patterns returned the correct answer. The difference is entirely in cost, not accuracy — on this task.
The model also spent 391 reasoning tokens chewing through the raw dataset in the naive run versus 10 when handed the pre-aggregated result.
Caching note (from the artifact): the naive run reported cached_tokens: 6912 of its 7,462 input tokens, so most of that input billed at the cheaper cached rate. The token volume gap (98%) therefore overstates the dollar gap. We report tokens processed, not cost; any cost figure must apply cached-vs-fresh per-Mtok rates.

Verified source facts about Claude PTC (not measured here)

From the Claude programmatic-tool-calling docs (fetched 2026-07-13):

Enable by adding code_execution to the tools list and allowed_callers: ["code_execution_20260120"] to each tool Claude may invoke from code. code_execution_20260120 and code_execution_20260521 are interchangeable in allowed_callers.
Requires code_execution_20260120 or later. Supported models listed: Claude Fable 5, Mythos 5, Opus 4.8/4.7/4.6/4.5, Sonnet 4.6/4.5.
Available on the Claude API, Claude Platform on AWS, and Microsoft Foundry (Hosted-on-Anthropic deployment). Not on Amazon Bedrock or Google Cloud at time of writing.
Not eligible for Zero Data Retention (ZDR).
Anthropic's published benchmark figure: on BrowseComp and DeepSearchQA, adding PTC improved performance ~11% while using 24% fewer input tokens. (This is the realistic multi-step-search figure — much smaller than our single-task illustration, because those tasks don't dump one giant blob into context.)

Limitations

This is a single-task illustration, deliberately shaped so the entire intermediate dataset sits in context. It shows the mechanic at its most favorable. Real workloads land closer to Anthropic's 24%-input-token figure, or anywhere in between, depending on how large intermediate results are relative to the rest of the prompt.
Measured on OpenAI gpt-5.5-2026-04-23, not on Claude PTC. Token counts, tokenizer, and pricing differ across providers and models.
No production benchmark, no user study, no claim that any customer pipeline was migrated.
Prices change; any cost figure derived from these token counts should be recomputed against current per-Mtok rates.