Gpt 5 6 Programmatic Tool Calling Multiagent Token Proof 2026

Date: 2026-07-15
Track: api-backed-poc
Model: gpt-5.6-luna (OpenAI Responses API)
Harness: scripts/gpt56-ptc-token-proof.py (custom, dependency-free, budget-guarded via scripts/proof_budget.py)
Artifact: data/lab-runs/gpt-5-6-programmatic-tool-calling-multiagent-token-proof-2026.openai.json

Purpose

Measure, on the real GPT-5.6 API, how programmatic tool calling (PTC) changes token cost and model round-trips versus classic function calling — and find the condition that decides which one is cheaper. No secrets, no customer data; both tasks are synthetic and deterministic.

What programmatic tool calling is

With classic function calling, every tool result the model requests is sent back into the model's context window, and you pay for those tokens. With PTC, the model instead writes JavaScript that runs in a hosted V8 runtime (no network, no filesystem, no persistent state), calls your tools from inside that runtime, filters/aggregates the results there, and returns only the final answer to the model's context. You enable it by adding a {"type": "programmatic_tool_calling"} tool and setting allowed_callers: ["programmatic"] on the functions the program may call. The response then carries program, program-issued function_call, and program_output items whose call_id/caller linkage must be preserved.

Setup

Two deterministic scenarios, each run once per mode (classic vs PTC), same model tier:

Tiny outputs — get_metric(id) returns a single integer (id*id). Task: fetch ids 1..8 and sum. Ground truth = 204. Each tool result is a few tokens.
Bulky outputs — get_orders(day) returns 40 order rows (id, customer, region, amount, currency, refunded flag, note). Task: across days 1..8, total amount where refunded is true. Ground truth = 52584. Each tool result is a large JSON payload; classic mode drags all 320 rows through the model context, PTC keeps them in the runtime.

Reproduce:

python3 scripts/gpt56-ptc-token-proof.py

The script runs a manual tool loop, records real API usage per turn, and writes the JSON artifact. Budget is checked before every call and recorded in data/state/openai-token-usage.json.

Results (real API usage counters)

Scenario 1 — tiny tool outputs

Metric	Classic function calling	Programmatic tool calling	Change
Model round-trips	2	9	+7
Tool calls executed	8	8	—
Output tokens	155	91	−41%
Total tokens	645	1,392	+116% (2.2× more)
Answer	204	204	both correct

Scenario 2 — bulky tool outputs (320 rows)

Metric	Classic function calling	Programmatic tool calling	Change
Model round-trips	2	9	+7
Tool calls executed	8	8	—
Output tokens	252	126	−50%
Total tokens	19,491	1,496	−92.3% (13× fewer)
Answer	52584	52584	both correct

What this shows

PTC is not universally cheaper. The deciding variable is the size of the intermediate tool outputs.
With tiny results, PTC's overhead (writing the program, plus the runtime awaiting each client-side tool call one at a time) made it cost 2.2× more total tokens on this task.
With bulky results, PTC kept all 320 rows inside the runtime and returned only the total, cutting total tokens ~92% (from 19,491 to 1,496).
The PTC round-trip count (9 vs 2) is higher because our tools are client-side: the hosted program still pauses to let our app run each get_* call. PTC's savings come from keeping tool outputs out of the model context, not from cutting network round-trips to your app.
Both modes returned the correct answer in both scenarios.

Limitations

Two synthetic tasks, one run per mode per scenario, one model tier (gpt-5.6-luna). This is a bounded lab check, not a benchmark or statistical study. Counts will vary with the model's generated program, tool-output size, reasoning effort, and whether tools run server-side (hosted/MCP) or client-side. No production system was migrated and no customer data was used. Pricing, latency, and multi-agent (Ultra/beta) behavior were not measured here.