Your LLM Bill Is an Architecture Problem, Not a Prompt Problem

The real bottleneck: you cannot see who spent the money#

The month your AI product starts working is the month the invoice stops making sense. Usage grew, so the bill grew. That part is fine. What is not fine is the next question from your board or your CFO, which is some version of: which customers are we losing money on?

Most teams cannot answer it, and the reason is structural rather than lazy. Provider billing is organised around credentials, not around your business. OpenAI's Costs endpoint is https://api.openai.com/v1/organization/costs, it needs an Admin key, and it buckets spend by day, with 1d currently the only supported bucket_width. You can group the result by line item, by project, and — since the API-key dimension was added — by key. What you cannot do is ask it for cost per tenant, cost per feature, or cost per support ticket resolved, because the provider has never been told those things exist.

So the bill arrives as one number. Inside that number sit three separate business problems that happen to share a payment method.

Cost. Your gross margin per customer is unknown, which means your pricing is a guess. Founders discover this at the worst moment, usually while negotiating an enterprise contract with a customer whose usage pattern turns out to be six times the average.

Reliability. Rate limits are enforced per organisation tier, so one runaway internal batch job can push your paying customers into 429s. Without attribution you learn this from a support ticket rather than a dashboard.

Latency. Prompt caching is sold on two benefits at once. Anthropic's pricing documentation describes it as reducing "costs and latency by reusing previously processed portions of your prompt across API calls." When your cache hit rate quietly collapses, the first symptom users notice is a slower product, and the bill confirms it two weeks later.

One missing capability causes all three. Not a missing prompt technique — a missing dimension in your own telemetry.

Why the naive in-prompt fix fails#

The reflex response is to shorten the prompt. Trim the system message, drop a few examples, tell everyone to be concise. It feels like the cheapest possible intervention. It fails for four reasons, and each one is documented by the vendors themselves.

Trimming can switch caching off entirely. Both providers enforce a minimum cacheable length. OpenAI states that "the minimum cacheable prompt length is 1,024 tokens for GPT-5.6 and later and 2,048 tokens for models older than GPT-5.6." Anthropic's minimums are per model: 512 tokens for Claude Opus 5, 1,024 for Sonnet 5 and Opus 4.8, and 4,096 for Haiku 4.5. Trim a prompt from 1,200 tokens to 900 on a model with a 1,024-token minimum and the prefix silently stops caching. Nothing errors. The bill goes up while the prompt got shorter.

One byte in the wrong place costs you the whole cache. Caching is a prefix match. OpenAI puts it plainly: "Cache reuse requires the entire rendered prefix to match. If content or a relevant setting changes before a breakpoint, the prefix after that change cannot match the existing cache entry." A timestamp, a request ID, or a randomly ordered JSON key in your system prompt invalidates everything after it, on every single request. This is the single most common cause of a cache hit rate that reads zero.

Prompt length does not touch the expensive half. On Claude Sonnet 5 the list price is $2 per million input tokens and $10 per million output tokens. Output is five times the price of input. A team that spends a sprint shaving input tokens while leaving response length uncontrolled has optimised the cheap side of the invoice.

And none of it produces a number you can act on. Even a successful prompt diet leaves you with the same single blended figure. You still cannot say which tenant is unprofitable, so you still cannot price, cap, or upsell.

Prompt work is a control. Attribution is the instrument that tells you which control to pull. Buying controls before instruments is how teams end up with a cache, a router, and a bill nobody can explain.

Three attribution patterns you can ship in a week#

These are ordered by effort. Most teams need the first two; the third matters once finance needs the numbers to reconcile against an actual invoice.

Pattern 1 — a key or project per surface#

Issue a separate API key for each meaningful surface: the customer-facing chat, the nightly enrichment job, the internal eval harness, the demo environment. Then group provider costs by key. This is one afternoon of work and it immediately separates "customers cost us money" from "our own batch jobs cost us money," which is usually the first real surprise.

The limit is granularity. Credentials are coarse. You will not get per-tenant numbers this way unless you are willing to run per-tenant keys, which becomes an operational burden well before a hundred customers. Treat this as the floor, not the destination.

Pattern 2 — a gateway ledger built from usage fields#

Route every model call through one internal service. That service attaches your business dimensions (tenant ID, feature, request ID, plan tier), makes the call, and writes one ledger row per response using the token counts the API already returns.

Those counts are the part people miss. Anthropic returns three distinct input figures: cache_creation_input_tokens ("number of tokens written to the cache when creating a new entry"), cache_read_input_tokens ("number of tokens retrieved from the cache for this request"), and input_tokens ("number of input tokens which were not read from or used to create a cache"). Multiply each by its own rate and you have exact, per-request cost — not an estimate — at whatever business granularity you chose to log.

This is the pattern that answers the CFO's question. It is also the one that makes the next section's controls measurable, because a cache that stops working shows up as cache_read_input_tokens going to zero on a specific tenant, on a specific day.

Pattern 3 — reconciliation against the provider's own numbers#

Once a day, pull the provider's cost buckets and compare the total against your ledger. On OpenAI that is the Costs endpoint described above; the daily bucket is a hard floor on time granularity, so reconcile daily and do not expect hourly truth. A persistent gap between your ledger and the invoice means you have calls escaping the gateway — a script, a notebook, a vendor integration — and that gap is worth finding before it becomes a line item.

A useful discipline here: OpenAI's prompt_cache_key parameter exists for "grouping and distribution during higher-volume traffic," so the key you choose for cache grouping and the dimension you attribute cost to should be the same string. Two different taxonomies for the same traffic is how reconciliation stops converging.

The controls, and what each one is actually for#

With a ledger in place, three architectural controls become tunable rather than aspirational. They are not interchangeable. Each one pays on a different traffic shape, which is precisely why attribution has to come first.

Server-side pruning is for agent loops, where tool results accumulate until the context is mostly transcript. Anthropic's context editing beta (header context-management-2025-06-27) applies a clear_tool_uses_20250919 strategy that clears old tool results at a configurable trigger, defaulting to 100,000 input tokens, while preserving the most recent keep tool uses, defaulting to 3. The response reports cleared_input_tokens, so the saving is measured rather than assumed. On a simple request-and-response product this control does nothing at all.

Prompt caching is for any workload with a large stable prefix. The order the request renders in is tools, then system, then messages, so stable content goes first and volatile content goes after the last breakpoint. Anthropic allows up to 4 cache breakpoints. The TTL choice has an explicit break-even: 5-minute cache writes cost 1.25x base input price and 1-hour writes cost 2x, against cache reads at 0.1x, which Anthropic spells out as caching that "pays off after one cache read for the 5-minute duration (1.25x write), or after two cache reads for the 1-hour duration (2x write)." Our measured comparison of semantic and prompt caching covers the case where a second cache layer is worth adding on top. A ledger built on token counts also has to survive a model migration, because a new tokenizer changes the count without changing the price — the Sonnet 5 break-even model works that case through.

Hybrid routing is for workloads with genuinely different task shapes underneath one product surface. Classification, extraction, and short structured replies can run on a cheaper tier while open-ended reasoning stays on the frontier model. We have written up how routing performs against a measured cost baseline separately. The trap is routing before caching, which the numbers below make concrete.

What the money actually does: a worked model#

The figures below are computed from published list prices, using illustrative traffic parameters. They are a model, not a measurement of any Effloow system or client. Swap in your own numbers from your own ledger — that substitution is the entire point.

Parameters: 300,000 requests per month; 4,800-token static prefix; 1,200 variable input tokens; 400 output tokens. Prices are Claude Sonnet 5 at $2 per MTok input and $10 output, with 5-minute cache writes at $2.50 and cache reads at $0.20, and Claude Haiku 4.5 at $1 and $5, with cache writes at $1.25 and reads at $0.10.

The caching row breaks down as 1,296 MTok of cache reads at $0.20 ($259.20), 144 MTok of cache writes at $2.50 ($360.00), and 360 MTok of uncached variable input at $2 ($720.00).

Two things in that table are worth more than the headline percentage.

First, caching removes 47% of the bill and routing adds only another 6.6 points. That is not an argument against routing; it is arithmetic. Caching had already collapsed the input side to a tenth of list price, so moving traffic to a model that is half the price of Sonnet 5 has much less left to work on. A team that builds a router first, on the theory that cheaper models are the obvious win, does the harder engineering for the smaller number.

Second, output cost barely moves in either row. At a 5:1 output-to-input price ratio, response length control — structured outputs, explicit length limits, dropping a verbose format — is often the third lever nobody scheduled. Our token optimization guide goes into that side.

Can this survive your workflow?#

Do it if: you serve multiple customers or multiple features from one credential; your monthly LLM spend is large enough that a 40% swing changes a hiring decision; you are about to set or defend prices; or someone has asked for gross margin per customer and you had to guess.

Skip it if: you run a single-tenant internal tool where the whole bill is a rounding error, or you are pre-launch and the honest answer is that traffic shape is still unknown. Building a gateway before you have traffic to attribute is premature. A key per surface (Pattern 1) is enough at that stage and costs an afternoon.

The founder's ROI question. Payback is the build cost divided by the monthly saving. In the model above, moving from baseline to cached-and-routed saves $2,578.20 per month. Price the work at your own loaded engineering day rate: a week of one engineer at $600 per day is $3,000, which pays back in about five weeks and then compounds every month that traffic grows. What the model deliberately does not include is the second-order return, which is usually larger — being able to price a contract with the actual per-tenant cost in front of you, instead of a blended average that hides your worst customer.

For your engineers#

Implementation notes that save a debugging cycle:

• Verify caching is live before trusting any saving. If cache_read_input_tokens is zero across repeated identical-prefix requests, something in the prefix is varying — a clock, a UUID, unsorted JSON, or a tool list whose order is not deterministic.
• Keep the tool array in a fixed order. Tools render before the system prompt, so a reordered tool list invalidates every downstream cache entry.
• Compute cost per request from the three usage fields separately rather than summing all input tokens. Cache reads, cache writes, and uncached input have three different rates; a single blended multiplier will under- or over-report by a wide margin exactly when hit rates change.
• Reconcile daily, not hourly. bucket_width on OpenAI's Costs endpoint supports 1d, so hourly reconciliation will not line up.
• Use one identifier for both cache grouping and cost attribution. Two taxonomies over the same traffic guarantees a permanent unexplained gap.
• Prune only what you measured. clear_at_least interacts with prompt cache decisions, so aggressive clearing can invalidate a cached prefix and cost more than it saves.

What Effloow added#

The vendors publish the mechanics; none of them publish the interaction. This article contributes three things beyond the source documents: the ordering argument that attribution is an instrument and caching, routing, and pruning are controls that cannot be chosen without it; a list-price cost model showing that caching captures 47 points and routing adds 6.6 more on the same traffic, which reverses the usual build order; and the three attribution patterns ranked by effort against what each one can and cannot answer.

The model is arithmetic on published prices with stated assumptions. Applied to your traffic, the same arithmetic produces a different answer — that is the useful part.

If you want that answer measured on your own stack and written up as a claim-bound evidence report rather than an estimate, that is what Proof Studio covers: one specific claim, executed, with the method, the failures, and the cost published alongside the result. Tell us the claim and the workload at /contact and we will reply by email with a scope.

Sources#

• Anthropic, Pricing — model list prices, cache write and read multipliers, batch discount
• Anthropic, Prompt caching — minimum cacheable lengths, breakpoint limit, usage field definitions
• Anthropic, Context editing — beta header, clear_tool_uses_20250919 strategy, trigger and keep defaults
• OpenAI, Prompt caching — minimum cacheable prompt length, prefix match rule, prompt_cache_key
• OpenAI, Costs API reference — grouping dimensions including api_key_id
• OpenAI, Usage and Costs API cookbook — endpoint, admin key requirement, bucket_width support