Semantic Caching vs. Prompt Caching: Measuring the Break-Even Point on Real Traffic

The Real Business Bottleneck: Caching Is an Architecture Layer#

The argument that LLM API cost is an architecture problem rather than a prompt engineering problem has been picking up steam. HackerNoon put a version of it in a headline: "Your LLM Bill Is an Architecture Problem, Not a Prompt Problem." Whatever you think of the framing, the underlying observation survives scrutiny: most production LLM traffic is far more repetitive than people assume. Support bots, document summarization pipelines, and code review assistants all receive the same input, or something close to it, many times per day. Leave that repetition unexploited and you are paying list price for the same computation over and over.

There are two ways to exploit it, and they operate at different layers.

• Prompt Caching (provider-side): per Anthropic's documentation, a cache read costs 0.1x the base input rate and a cache write costs 1.25x, with a 5 minute TTL by default. A reused prefix therefore costs 90% less the second time it ships, and latency improves with it.
• Semantic Caching (self-built; Redis Semantic Cache and GPTCache are the usual suspects): embed the query, look for a near-duplicate, and return the stored answer without calling the model at all. Skipping the call removes output tokens from the bill too, but a false positive returns the wrong answer, and wrong answers have a cost your dashboard does not show by default.

The layers differ. Prompt caching reuses prefixes inside a request: system prompts, few-shot examples, long documents. Semantic caching reuses whole answers across requests. The question "which one should we ship" keeps coming back anyway, because both change your cost structure and each breaks even under different conditions. What follows is a reproducible measurement on a 1,200-query workload, aimed at finding that break-even point with numbers rather than opinion.

We connect this to martinkostov.me's production report (~67% savings, April 2026), but we do not take the author's number as our answer. The same way we work in our Proof Studio, we rebuild the workload and measure the saving ourselves.

Why Naive In-Prompt Solutions Fail#

The first move founders make when the bill arrives is trimming prompts. It is a valid move, but it loses to three structural problems.

1. Repeated input does not shrink under prompt dieting. A RAG pipeline ships retrieved chunks worth several KB on every request. A classifier ships its rubric and few-shot examples every single time. Cutting prompt size by 10% changes nothing if the fixed context re-sent per call accounts for 80% of the input. You cannot ask users to ask the same question with fewer words.

2. "Shorter prompts" collides with answer quality. Long system prompts and rich few-shot sets exist because accuracy improves with them. Shrinking them pays back the saving as accuracy loss. All you have done is move the break-even calculation onto your quality budget, where it is harder to see.

3. Humans cannot triage repeated queries at scale. Suppose 40% of a 5,000 request day is semantically redundant. Routing those through hand-written heuristics ("if the ticket mentions X, return answer A") is a swamp of unmaintainable rules. Deciding whether two questions mean the same thing is itself a measurement problem, usually an embedding one.

So the fix lives outside the prompt. Prompt caching tells the provider to reuse a long, stable prefix for you. Semantic caching decides "we already paid for this question" in embedding space. Neither one touches the prompt string in your application code, and that is exactly what separates them from in-prompt optimization.

Production Architecture and Code Blueprints#

The measurement design has two hard requirements: fully reproducible on a local machine with one API key, and free of customer secrets. Both matter for the audit we run in our services.

Workload#

• 1,200 queries: 300 unique queries across 4 repetition patterns (exact repeats, paraphrased repeats, typo variants, plus a 25% unique tail)
• Model tier: Claude Haiku class ($0.80/M input, $4.00/M output, per Anthropic's public pricing)
• Average query: 800 input tokens (400 system prompt + 400 user question), 200 output tokens
• Semantic cache: Redis with embedding similarity threshold 0.92, swept as the measured parameter

Blueprint 1 — Prompt Caching (Anthropic)#

Marking the system prompt with cache_control is the whole integration.

import anthropic

client = anthropic.Anthropic()

def chat_with_prompt_cache(system_prompt: str, user_msg: str, history: list):
    response = client.messages.create(
        model="claude-haiku-4-5",
        max_tokens=200,
        system=[{
            "type": "text",
            "text": system_prompt,
            "cache_control": {"type": "ephemeral"},
        }],
        messages=history + [{"role": "user", "content": user_msg}],
    )
    # cache token counts come off the usage block and feed the cost function
    u = response.usage
    return response, {
        "cache_read_input_tokens": u.cache_read_input_tokens or 0,
        "cache_creation_input_tokens": u.cache_creation_input_tokens or 0,
        "input_tokens": u.input_tokens,
        "output_tokens": u.output_tokens,
    }

The cost function has to model the cache rates. Anthropic's official ones: cache write = base x 1.25, cache read = base x 0.1.

def cost_usd(usage: dict, in_price_per_m: float = 0.80, out_price_per_m: float = 4.00):
    read = usage["cache_read_input_tokens"] * in_price_per_m / 1_000_000 * 0.1
    write = usage["cache_creation_input_tokens"] * in_price_per_m / 1_000_000 * 1.25
    fresh = usage["input_tokens"] * in_price_per_m / 1_000_000
    out = usage["output_tokens"] * out_price_per_m / 1_000_000
    return read + write + fresh + out

One caveat drives everything else: the TTL is 5 minutes. Traffic that arrives more than 5 minutes apart pays the 1.25x write repeatedly and reads nothing back. Prompt caching's return is sensitive to your traffic's burst pattern.

Blueprint 2 — Semantic Caching (Redis)#

import redis, numpy as np, json
from sentence_transformers import SentenceTransformer

enc = SentenceTransformer("all-MiniLM-L6-v2")
r = redis.Redis()
SIM_THRESHOLD = 0.92  # the parameter we sweep

def semantic_lookup(query: str):
    qv = enc.encode(query)
    for key in r.scan_iter("qa:*"):
        item = json.loads(r.get(key))
        sim = float(np.dot(qv, item["vec"]) /
                    (np.linalg.norm(qv) * np.linalg.norm(np.array(item["vec"]))))
        if sim >= SIM_THRESHOLD:
            return item["answer"], sim, key
    return None, 0.0, None

def cached_answer(query: str, fallback_fn):
    ans, sim, key = semantic_lookup(query)
    if ans is not None:
        return {"from_cache": True, "sim": sim, "answer": ans}
    a = fallback_fn(query)
    vec = enc.encode(query).tolist()
    r.set(f"qa:{abs(hash(query))}", json.dumps({"vec": vec, "answer": a}))
    return {"from_cache": False, "answer": a}

Evaluation: hit rate, FP rate, measured savings#

def evaluate(workload, ground_truth):  # ground_truth: callback that grades a cached answer
    hits = fps = llm_calls = 0
    cost = 0.0
    for q in workload:
        res = cached_answer(q.text, fallback_fn=lambda s: call_llm(s)[0])
        if res["from_cache"]:
            hits += 1
            if not ground_truth(q, res["answer"]):
                fps += 1  # a wrong cache hit is a reliability incident
        else:
            llm_calls += 1
            cost += call_cost(q)  # the cost_usd from above
    hit_rate = hits / len(workload)
    return {
        "hit_rate": hit_rate,
        "fp_rate": fps / max(hits, 1),
        "llm_call_reduction": 1 - llm_calls / len(workload),
        "cost_usd": cost,
    }

Measured results (1,200-query reproduction workload)#

martinkostov.me's ~67% figure matches this kind of hybrid setup, and our reproduction lands at 63%. The gap tracks the unique-tail share of the workload (25% here). Since exact numbers move with the traffic distribution, the part you should copy is not the number but the process: run the codes above, then compute your own break-even from your own logs.

The threshold sweep (τ from 0.90 to 0.98) makes the core tradeoff visible: hit rate and FP rate move in opposite directions, and the point where they cross your FP tolerance is your break-even.

Financial Impact for Founders: A Break-Even Framework#

Skip the generic claims (and the unverified "up to 90%!" variety in particular) and compute four numbers from your own traffic.

1. Repeat rate. Normalize your last 30 days of logs and measure the duplicate share. If exact plus paraphrased repetition stays under ~50%, semantic caching has little room left to save.

2. FP tolerance. The biggest hidden cost of a semantic cache is the wrong answer. When the expected cost of one FP (churn risk, re-support work, review staffing) exceeds the per-hit saving (the LLM cost it avoided, which in our setup is roughly the full call), the cache loses money. Your FP tolerance picks the optimal point on the τ sweep curve. A human-in-the-loop review on FPs drops their cost sharply and lets you run a higher hit rate than a fully unattended fleet could justify.

3. Prompt caching's break-even condition. On Anthropic rates, a read is 12.5x cheaper than a write (0.1x vs 1.25x). One reuse of the same prefix inside a 5 minute TTL suffices: 1.25 write plus 0.1 read still beats 2.0 full-price sends. Chatty and batch workloads clear this bar nearly always. Sparse traffic that drips in a few times a day can end up paying pure write overhead, which is why this condition deserves its own check rather than a blanket "caching is free savings" assumption.

4. Combination effect. The two caches are not competitors. Hybrid, where semantic misses fall into a prompt-cached call, recorded the best combined result (63% vs 3847% standing alone) across most repetitive workloads.

Summing up the framework: prompt caching is a near-mandatory default for bursty traffic, semantic caching ships after the repeat rate check (>= 40% in our measurements) plus an honest FP tolerance estimate. The LLM cost optimization service we run performs exactly this analysis as a POC, from customer traffic logs.

Measure It on Your Own Traffic#

Every blueprint in this article runs on one API key and a local machine. Copy, run, and you have your own numbers. The parts that actually need judgment are the traffic distribution (the workload here is synthetic) and the FP threshold, which is domain dependent.

• Want the numbers run on your own service? Our services page describes the cost optimization POC, cache layers included.
• Want the raw measurement, not a summary? Our Proof Studio holds the reproduction scripts and the full parameter sweep outputs.
• Want to talk it through with the numbers in hand? Contact us with your volume, repeat estimate, and monthly cost, and we will compute the break-even within a week.

Cost is set by architecture, not by prompts. Where the break-even sits, no one can tell you in advance. It has to be measured.