Beyond Naive Summarization: Context Compaction with Temporal Memory

If your AI assistant, copilot, or support agent holds conversations longer than a handful of turns, you are paying for the same words again and again. Every turn, the model re-reads everything that came before: the greeting from Monday, the tool output from twenty minutes ago, the correction the user already made twice. The invoice grows with the square of the conversation length, and the product gets slower and less accurate at exactly the moment your best customers use it most.

This guide covers the memory architecture that fixes it. The short version: history is not memory. A production AI product needs a server-side layer that decides what the model sees, keeps the prompt stable enough for caching to work, and extracts durable facts into a store that survives the session. Two open-source projects, Zep's Graphiti and Mem0, now make the extraction layer buildable in a week instead of a quarter, and the vendor pricing tables make the savings arithmetic public and checkable.

The Real Business Bottleneck: Cost, Reliability, Latency#

For a founder, "long conversations" is not a UX detail. It is a unit-economics problem with three faces.

Cost. LLM APIs bill input tokens on every request, and in a conversation the input is the entire accumulated history. A session where each turn adds 800 tokens costs almost nothing at turn 3 and re-reads roughly 40,000 tokens by turn 50. Summed over the whole session, that shape is quadratic: the 100-turn version of that conversation bills about 4 million input tokens, and 98% of them are repeats. Your longest sessions come from your most engaged customers, so the users who like the product most are the ones with the worst gross margin.

Reliability of the answer itself. Longer context is not just more expensive, it is measurably less accurate. The widely cited "Lost in the Middle" study (Liu et al., 2023) found that model performance degrades significantly when the relevant information sits in the middle of a long context rather than at the beginning or end. In a 60-turn support conversation, the constraint the user stated in turn 4 is precisely the thing a long-context model is most likely to miss by turn 55. Customers experience this as the assistant "forgetting," and it erodes trust faster than any latency problem.

Latency. Time-to-first-token grows with input size, because the model must ingest the whole prompt before producing anything. A product that felt snappy in the demo (short context) becomes sluggish for exactly the power users whose sessions are longest. And when the history no longer fits the context window at all, teams bolt on emergency truncation that silently drops instructions, which is how billing bots forget refund limits.

The bottleneck, in other words, is not the context window size. Windows of 200K tokens and more exist; filling them is the mistake. The bottleneck is the absence of any layer in your stack that separates what happened (history) from what the model needs to know right now (memory).

Why Naive In-Prompt Solutions Fail#

Most teams attack this from inside the prompt first. Three popular moves, and why each one stalls:

"Keep the last N turns." A sliding window caps cost, but it forgets by position, not by importance. The user's account tier, stated in turn 2, falls out of the window at turn 22; the small talk from turn 21 stays. There is no correlation between how recently something was said and how much it matters, so a sliding window guarantees the assistant eventually contradicts something the user already told it.

"Have the model summarize the conversation so far." In-prompt summarization looks like compression but behaves like lossy rewriting. A summary is a paraphrase produced by a model with no stake in which details survive; constraints, negations, and changes of state are exactly what paraphrase flattens. "The user wanted the Pro plan, then downgraded to Starter after seeing the price" routinely compresses to "user interested in Pro plan." The summary also silently loses time: it records what was said but not what is still true.

"Summarize every few turns to keep the prompt small." This variant adds a second failure: it destroys your cache. Both major vendors bill cached input at a fraction of the normal rate, but only when the prompt prefix is byte-stable between requests. A summarizer that rewrites the head of the conversation every five turns invalidates the cache on every rewrite, so the team pays full price for input tokens while believing they have optimized. The two techniques fight each other unless the architecture is designed so they don't.

The common failure in all three: they treat memory as a prompt-writing problem when it is a server-side data problem. What the model should see on each turn is a query result, not an archive.

Production Architecture: Three Layers, in Order#

Ship these in order. Each layer works alone; together they compound.

Layer 1: Server-Side Pruning with a Stable Head#

Before any request leaves your server, deterministic code decides what the model sees. The one design rule that makes the next layer (caching) work: everything stable goes first, everything volatile goes last, and nothing in the head ever gets rewritten.

MAX_TOOL_RESULT_CHARS = 4_000
RECENT_TURNS = 10

def build_context(system_prompt, memory_block, turns):
    # Head: byte-stable across turns -> cacheable
    messages = [{"role": "system", "content": system_prompt}]
    # Middle: replaced wholesale when memory updates (rare), never edited in place
    if memory_block:
        messages.append({"role": "system",
                         "content": f"Known facts about this user/session:\n{memory_block}"})
    # Tail: volatile recent window
    for t in turns[-RECENT_TURNS:]:
        content = t.content
        if t.role == "tool" and len(content) > MAX_TOOL_RESULT_CHARS:
            content = content[:MAX_TOOL_RESULT_CHARS] + "\n[truncated by server]"
        messages.append({"role": t.role, "content": content})
    return messages

The point is not sophistication. It is that the decision happens in your infrastructure, on every request, deterministically, instead of depending on a model's discretion or a prompt author's discipline. (For the static trimming techniques that complement this, see our token optimization guide.)

Layer 2: Prompt Caching on the Stable Prefix#

With a stable head, caching turns from a checkbox into real money. The vendor numbers are public:

Read that table against Layer 1 and the design rule becomes cash: every token that sits in a stable prefix costs one-tenth of a token that doesn't. A 6,000-token system prompt plus memory block, re-read 50 times in a session, bills like 30,000 tokens instead of 300,000, but only if your server never rewrites it mid-session. This is the quantitative reason naive periodic summarization is a false economy.

Layer 3: Temporal Memory Extraction#

Layers 1 and 2 manage the current session. The third layer answers the harder question: what does the assistant know at turn 1 of session 30, three weeks after session 29? Resending history across sessions is out of the question; a summary-of-summaries degrades like a photocopied photocopy. The production pattern, established by the MemGPT paper's OS-inspired memory hierarchy (Packer et al., arXiv:2310.08560) and now shipped by two open-source projects, is extraction: after each turn, a background process pulls durable facts out of the dialogue and writes them to a queryable store. At request time, the server retrieves only the facts relevant to the current turn and injects them into the memory block from Layer 1.

The differentiator to evaluate is how the store handles facts that change:

• Graphiti (Zep, Apache-2.0) builds a temporal knowledge graph with bi-temporal tracking: when a fact was true in the world versus when the system learned it. When new information supersedes old ("the user switched from the Pro plan to Starter"), the old edge is invalidated with a validity window, not deleted, so the system can answer both "what plan is the user on?" and "what plan were they on in June?". Retrieval is hybrid (semantic embeddings + BM25 keyword + graph traversal) and does not require an LLM call at query time; the project documents typical sub-second query latency, with Zep's managed platform claiming sub-200ms at scale.
• Mem0 (Apache-2.0) runs extraction and consolidation over a hybrid vector/graph store. Its self-reported benchmarks (April 2026 algorithm, measured on the managed platform) score 92.5 on LoCoMo and 94.4 on LongMemEval while holding retrieved context near 7K tokens with p50 retrieval latency under 1.1 seconds. Those are Mem0's own published figures, and the project itself notes open-source deployments should expect directionally similar rather than identical numbers. Still, the shape of the claim is the architecture's whole point: recall quality held while context stayed flat instead of growing with history.

Wiring it in is smaller than teams expect, because the pattern is asynchronous: extraction runs after the response is sent, off the latency path:

async def handle_turn(session, user_message):
    facts = memory.search(user_id=session.user_id, query=user_message, limit=10)
    context = build_context(SYSTEM_PROMPT, render_facts(facts), session.turns)
    reply = await llm.chat(context)
    # Off the critical path: extract/invalidate facts from this exchange
    background.enqueue(memory.add,
                       messages=[user_message, reply],
                       user_id=session.user_id)
    return reply

One integration note from running agentic systems in production: the extraction step is itself a workflow that can fail mid-write, and a memory store with half-applied updates is worse than no memory. Run it inside a durable execution layer (the same pattern we detail in our durable agent workflows guide).

Where routing fits#

Compaction and routing are independent multipliers on the same bill. Compaction shrinks how many tokens each request carries; hybrid model routing shrinks the price per token by sending easy requests to cheap models. If you gate traffic through a proxy layer such as LiteLLM, the memory block travels with the request regardless of which model serves it, so the two layers compose without coordination.

The Financial and ROI Impact#

The arithmetic is checkable from the pricing tables above. Take a support copilot with sessions averaging 100 turns at 800 tokens of new content per turn, on a model billing $5 per million input tokens (Claude Opus 5's published base rate):

• Raw accumulation: each turn resends everything, so the session bills the triangular sum: roughly 4.0M input tokens, about $20.20 per session before output tokens.
• Compaction + extraction: each turn carries a ~6K stable prefix (cached at 0.1×) plus a ~8K volatile tail, so the session bills roughly 0.8M full-price tokens and 0.3M cache-read tokens, about $4.15 per session.

That is roughly a 5× reduction on input spend from architecture alone, before routing multiplies it further. These are illustrative figures computed from the stated assumptions and published prices, not a measured benchmark. Your ratio depends on session length and tool verbosity; the quadratic-versus-flat shape does not. Run your own traffic profile through the same arithmetic before committing.

The second-order returns matter as much as the invoice. Flat context means flat time-to-first-token, so the product stays fast for heavy users. Facts retrieved by relevance rather than recency mean fewer "you already told it that" moments, which is a retention lever no prompt tweak reaches. And cross-session memory is a feature customers can see and pay for, funded by an architecture change that cuts costs.

Can this survive your workflow? Before adopting, answer four questions:

1. Do your sessions actually run long (15+ turns) or recur across visits? If not, Layers 1–2 are enough; skip extraction.
2. Can you tolerate eventually-consistent memory? Extraction is asynchronous; a fact from turn N is reliably available at turn N+2, not always N+1.
3. Do facts in your domain change (plans, addresses, preferences)? If yes, temporal invalidation (Graphiti's model) matters more than raw recall scores.
4. Who audits what the assistant "knows"? A memory store needs the same deletion, export, and inspection story as any other user-data store. Plan for it before, not after, your first enterprise security review.

When to use / when to skip. Use this architecture for assistants, copilots, CRM and support products with multi-session relationships or long agentic sessions. Skip it for stateless single-shot features (translation, one-off extraction, document Q&A over static corpora; plain RAG serves those better), and skip Layer 3 while you are pre-product-market-fit with sessions under a dozen turns: a sliding window's flaws are tolerable at that scale and the ops surface of a graph store is not free.

Get the Architecture Before the Invoice Forces It#

Every team that ships a conversational AI product eventually builds some version of this layer. The only question is whether it happens by design or after the first alarming invoice and the first "your bot forgot my order" escalation.

Effloow designs and builds this architecture for AI products: server-side context gates, cache-aligned prompt layouts, memory-store selection and integration (Graphiti, Mem0, or custom), and the measurement harness that proves the savings on your actual traffic instead of a blog post's assumptions. If your model spend is growing faster than your usage, or your assistant forgets things customers already said, see what we build and talk to us. A review of one week of your request logs is usually enough to size both the leak and the fix.

For Your Engineers#

What Effloow added: a cache-stability analysis showing why periodic in-prompt summarization and prompt caching are mutually defeating (the vendor pricing tables make this quantitative, not stylistic), a worked cost model with explicit assumptions for the quadratic-vs-flat comparison, and a four-question adoption checklist covering the consistency and compliance costs that memory-layer vendors' own docs underplay.

Primary sources. Graphiti repo and Zep docs for bi-temporal invalidation and hybrid retrieval; Mem0 repo for the LoCoMo/LongMemEval self-reported figures and their managed-platform caveat; MemGPT, arXiv:2310.08560 for the memory-hierarchy pattern; Lost in the Middle, arXiv:2307.03172 for positional degradation; Anthropic and OpenAI prompt-caching docs for the 0.1× cached-input pricing, write premiums, TTLs, and minimum cacheable lengths.

Implementation cautions. Anthropic cache writes cost 1.25× (5-minute TTL) or 2× (1-hour TTL) base input. A prefix that changes too often can cost more with caching enabled than without it; check the usage fields, since prompts under the per-model minimum (512–4,096 tokens) silently skip the cache. OpenAI's caching is automatic at ≥1,024 tokens but prefix-matched, so the same head-stability rule applies. Graphiti needs a graph database plus an LLM for extraction; budget for extraction-time model calls (they are small but nonzero and scale with message volume). Mem0's headline numbers are its own, measured on its managed platform. Treat them as the vendor's claim and benchmark recall on your domain's fact types, especially negations and state changes, before trusting the store in a billing-adjacent workflow. Neither store removes the need for Layer 1: retrieval output is still untrusted size-wise and needs the same server-side truncation gate as any tool result.