Date: 2026-07-10 · Runner: scripts/openai-lab-run.py · Provider/Model: OpenAI Responses API, gpt-5.5-2026-04-23

What we measured and why

Claude's web_search_20260318 adds a response_inclusion control, and OpenAI's hosted web_search tool adds return_token_budget. Both exist to stop verbose web-search results from piling up in the model context. Anthropic's own web-search docs state that "web search results retrieved throughout a conversation are counted as input tokens ... in search iterations executed during a single turn and in subsequent conversation turns."

Our bounded harness cannot invoke those hosted web-search tools directly, so we measured the underlying mechanism they target: the input-token cost of carrying three verbose "consumed" web-search result blocks in the model context versus keeping only their citations. Same follow-up question in both runs; only the retained context differs. Result blocks are synthetic placeholder text (no real facts) sized to resemble a scraped search snippet.

Measured result (real OpenAI usage.input_tokens)

• Input-token delta: 465 tokens saved (66.3% lower) on this single follow-up step.
• Baseline context billed 3.0x the trimmed context's input tokens.
• The trimmed run reached the same conclusion ("C") once given enough output budget, so dropping the raw blocks did not break the answer in this micro-run.

Limitations

Single synthetic micro-run: three fabricated result blocks, one follow-up, one model (gpt-5.5-2026-04-23). Not a benchmark, not a measurement of the actual response_inclusion or return_token_budget parameters, and not a billed-cost figure. The delta scales with how verbose the carried results are and how many turns they persist; do not read 66.3% as a fixed savings rate. Reproduce with the prompts in this note before quoting any number.

Effloow Lab OpenAI API Run: agentic-web-search-context-control-token-proof-2026

• Date: 2026-07-10T00:38:52.938598+00:00
• Model: gpt-5.5-2026-04-23
• Purpose: token-proof: input-token cost of retaining 3 consumed web-search result blocks in context (baseline)
• Request ID: 61565989-c5f0-42cb-9e82-0123722d50e8
• Usage: {"input_tokens": 701, "input_tokens_details": {"cache_write_tokens": 0, "cached_tokens": 0}, "output_tokens": 117, "output_tokens_details": {"reasoning_tokens": 92}, "total_tokens": 818}
• Prompt SHA-256: 77205a7846293dda9dcd6dee671c60494c533e0dadab9a89225680f3f529ec52
• Safety boundary: no confidential, customer, credential, or private data should be included in this run.

Prompt Excerpt

You are continuing a multi-step research session. Earlier in THIS SAME turn your web-search tool returned three raw result blocks. They have already been read and summarized. They are reproduced below in full because the client kept them in the model context (this simulates response_inclusion="full" / carrying consumed result blocks forward).

[WEB_SEARCH_RESULT 1] Title: "Grid-scale sodium-ion batteries reach parity — synthetic illustrative source A"
URL: https://example.com/sodium-ion-grid-2026
Content: A synthetic, illustrative writeup for a token-measurement lab. Sodium-ion cells are described here as an alternative grid-storage chemistry. The block repeats context so the raw result carries realistic bulk: deployment notes, cycle-life discussion, thermal behavior, supply-chain framing, regional pilot descriptions, cost-per-kilowatt-hour narrative, round-trip efficiency commentary, and integration caveats for utility operators. None of these figures are real; they exist only to give the result block representative length so that input-token accounting reflects a genuine consumed-result payload. The block continues with additional filler describing hypothetical pilot sites, maintenance intervals, warranty structures, inverter compatibility, and permitting timelines, again all synthetic and for measurement only, padded to approximate the size of a real scraped search snippet returned by an agentic search tool during one iteration of a research loop.

[WEB_SEARCH_RESULT 2] Title: "Second synthetic source B on long-duration storage economics"
URL: https://example.com/long-duration-storage-2026
Content: Another synthetic block for the lab. It discusses, in illustrative and non-factual terms, the economics of long-duration energy storage: capacity factors, arbitrage windows, capacity-market participation, degradation modeling, and the difference between power and energy sizing. The text is deliberately verbose to represent a realistic consumed search result. It adds pa

Output

C, because a standards-body style source is most relevant for procurement-grade compliance and safety requirements.

Limitations

This API run is a bounded lab check. It is not a production benchmark, user study, or proof that an external product works in a real customer environment.

Effloow Lab OpenAI API Run: agentic-web-search-context-control-token-proof-2026

• Date: 2026-07-10T00:38:57.150717+00:00
• Model: gpt-5.5-2026-04-23
• Purpose: token-proof: input-token cost with consumed result blocks dropped, citations retained (feature path)
• Request ID: 2355150d-4045-4ba7-b6a7-54286c622fdb
• Usage: {"input_tokens": 236, "input_tokens_details": {"cache_write_tokens": 0, "cached_tokens": 0}, "output_tokens": 120, "output_tokens_details": {"reasoning_tokens": 120}, "total_tokens": 356}
• Prompt SHA-256: 7da37cc3bdbe24081e0432c977572e7d1eb2aabdcad417e588bd09db0da39a03
• Safety boundary: no confidential, customer, credential, or private data should be included in this run.

Prompt Excerpt

You are continuing a multi-step research session. Earlier in THIS SAME turn your web-search tool returned three result blocks. They have already been read and summarized, and only their citations are retained in context (this simulates dropping consumed result blocks — the token-saving path that response_inclusion="excluded" and dynamic filtering target).

Retained citations only:
- [A] "Grid-scale sodium-ion batteries reach parity" — https://example.com/sodium-ion-grid-2026
- [B] "Long-duration storage economics" — https://example.com/long-duration-storage-2026
- [C] "Storage standards page" — https://example.org/storage-standards-2026

Follow-up task: In ONE short sentence, name which single source label (A, B, or C) a careful analyst would treat as most authoritative for a utility procurement decision, and why. Do not restate the sources.

Output

[NO OUTPUT TEXT]

Limitations

This API run is a bounded lab check. It is not a production benchmark, user study, or proof that an external product works in a real customer environment.

Effloow Lab OpenAI API Run: agentic-web-search-context-control-token-proof-2026

• Date: 2026-07-10T00:39:39.534671+00:00
• Model: gpt-5.5-2026-04-23
• Purpose: token-proof: parity check — trimmed/citations-only context with larger output cap to confirm the follow-up still answers
• Request ID: 99d3d74b-3f7c-42a5-b2c0-f3c1eebb37e5
• Usage: {"input_tokens": 236, "input_tokens_details": {"cache_write_tokens": 0, "cached_tokens": 0}, "output_tokens": 126, "output_tokens_details": {"reasoning_tokens": 99}, "total_tokens": 362}
• Prompt SHA-256: 7da37cc3bdbe24081e0432c977572e7d1eb2aabdcad417e588bd09db0da39a03
• Safety boundary: no confidential, customer, credential, or private data should be included in this run.

Prompt Excerpt

You are continuing a multi-step research session. Earlier in THIS SAME turn your web-search tool returned three result blocks. They have already been read and summarized, and only their citations are retained in context (this simulates dropping consumed result blocks — the token-saving path that response_inclusion="excluded" and dynamic filtering target).

Retained citations only:
- [A] "Grid-scale sodium-ion batteries reach parity" — https://example.com/sodium-ion-grid-2026
- [B] "Long-duration storage economics" — https://example.com/long-duration-storage-2026
- [C] "Storage standards page" — https://example.org/storage-standards-2026

Follow-up task: In ONE short sentence, name which single source label (A, B, or C) a careful analyst would treat as most authoritative for a utility procurement decision, and why. Do not restate the sources.

Output

C, because a standards page is most likely to provide procurement-relevant compliance requirements rather than market commentary.

Limitations

This API run is a bounded lab check. It is not a production benchmark, user study, or proof that an external product works in a real customer environment.