Skip to content
Effloow
← Back to article
EFFLOOW LAB LAB-RUN

Claude Programmatic Tool Calling Token Proof 2026

Evidence notes document the bounded local or source-based checks behind an Effloow article. They are not product endorsements, legal advice, or benchmark claims.
  • Date: 2026-07-13
  • Track: api-backed-poc
  • Purpose: measure, on a real model API, how many tokens a large intermediate tool result costs when it round-trips through the model context, versus when only the final aggregated result reaches context. This is the exact cost that Claude's programmatic tool calling is designed to remove.
  • Runtime for the measurement: OpenAI Responses API, model gpt-5.5-2026-04-23, via scripts/openai-lab-run.py.

Why measure this on OpenAI and not on Claude's PTC directly

Claude's programmatic tool calling (code_execution_20260120) keeps intermediate tool outputs inside the code-execution sandbox instead of passing them back through the model context. We did not run Claude's PTC feature here (it requires the code-execution tool version and a supported Claude model). Instead we isolated and measured the underlying mechanic — the token cost of putting a large intermediate payload into a model's context window — using a neutral reasoning-model API we can call within budget. The measured numbers below therefore quantify the cost PTC targets; they are not a benchmark of Claude's PTC implementation. Anthropic's own published figure for PTC is a separate, cited data point (see below).

Setup

Synthetic, deterministic scenario (no secrets, no customer data). An agent "fetched" 150 order-line records. Task: sum amount_usd for records where region == "EU" and status == "refunded". Ground truth (computed in plain Python): count=9, total_usd=5016.40.

Two prompts, identical task, differ only in how much of the intermediate data reaches the model:

  1. Naive pattern — the full 150-record JSON payload is placed in the model context; the model must filter and aggregate. Prompt file: naive_prompt.txt (SHA-256 prefix 8d230255a4689273).
  2. Programmatic pattern — an in-sandbox script already filtered and aggregated; only the compact result {"filtered_count":9,"total_usd":5016.4} reaches the model context. Prompt file: programmatic_prompt.txt (SHA-256 prefix 44f7c9b84b8027b6).

Command shape (run twice, once per prompt file):

python3 scripts/openai-lab-run.py --slug claude-programmatic-tool-calling-token-proof-2026 \
  --prompt-file /tmp/ptc-lab/naive_prompt.txt --max-output-tokens 500 \
  --output data/lab-runs/<slug>.naive.openai.json

Artifacts saved: *.naive.openai.json and *.programmatic.openai.json (each stores usage, request id, prompt SHA-256).

Results (real API usage counters)

Metric Naive (full data in context) Programmatic (compact result only) Change
Input tokens 7,462 120 −98.4% (62.2× fewer)
Output tokens 409 28 —
of which reasoning tokens 391 10 —
Total tokens 7,871 148 −98.1% (53.2× fewer)
Answer count=9, total_usd=5016.40 count=9, total_usd=5016.4 both correct
  • Naive request id: req_289602e69f1c49b6b5a7c9eb968e29df
  • Programmatic request id: req_68f375190756441bb66d215c45229c07
  • Both patterns returned the correct answer. The difference is entirely in cost, not accuracy — on this task.
  • The model also spent 391 reasoning tokens chewing through the raw dataset in the naive run versus 10 when handed the pre-aggregated result.
  • Caching note (from the artifact): the naive run reported cached_tokens: 6912 of its 7,462 input tokens, so most of that input billed at the cheaper cached rate. The token volume gap (98%) therefore overstates the dollar gap. We report tokens processed, not cost; any cost figure must apply cached-vs-fresh per-Mtok rates.

Verified source facts about Claude PTC (not measured here)

From the Claude programmatic-tool-calling docs (fetched 2026-07-13):

  • Enable by adding code_execution to the tools list and allowed_callers: ["code_execution_20260120"] to each tool Claude may invoke from code. code_execution_20260120 and code_execution_20260521 are interchangeable in allowed_callers.
  • Requires code_execution_20260120 or later. Supported models listed: Claude Fable 5, Mythos 5, Opus 4.8/4.7/4.6/4.5, Sonnet 4.6/4.5.
  • Available on the Claude API, Claude Platform on AWS, and Microsoft Foundry (Hosted-on-Anthropic deployment). Not on Amazon Bedrock or Google Cloud at time of writing.
  • Not eligible for Zero Data Retention (ZDR).
  • Anthropic's published benchmark figure: on BrowseComp and DeepSearchQA, adding PTC improved performance ~11% while using 24% fewer input tokens. (This is the realistic multi-step-search figure — much smaller than our single-task illustration, because those tasks don't dump one giant blob into context.)

Limitations

  • This is a single-task illustration, deliberately shaped so the entire intermediate dataset sits in context. It shows the mechanic at its most favorable. Real workloads land closer to Anthropic's 24%-input-token figure, or anywhere in between, depending on how large intermediate results are relative to the rest of the prompt.
  • Measured on OpenAI gpt-5.5-2026-04-23, not on Claude PTC. Token counts, tokenizer, and pricing differ across providers and models.
  • No production benchmark, no user study, no claim that any customer pipeline was migrated.
  • Prices change; any cost figure derived from these token counts should be recomputed against current per-Mtok rates.

Read the article

This note supports the public article and records what was actually checked.

Open article →