Why Agent While-Loops Fail in Production: Build Durable Workflows

Picture the demo that got your AI feature funded. An agent takes a customer request, thinks for a bit, calls a few tools, and returns a finished answer. It worked on stage because the whole run lived inside one healthy process for ninety seconds.

Now picture the same agent six weeks after launch. A customer asks it to process a refund, it charges the payment API, and then the container restarts during a deploy. The loop's memory is gone. There is no record of which step completed. When the customer retries, the agent runs the charge again. That is not a model-quality problem. It is an execution problem, and it is the most common way autonomous agents fail once real traffic arrives.

This guide is for founders and operators shipping agentic features into paying customers' workflows. It explains why the standard in-memory agent loop breaks, what a durable execution architecture looks like, and what the difference costs you in each direction. The two reference runtimes are Inngest AgentKit and Temporal, both of which document these patterns as first-class product features.

The Real Business Bottleneck: Runs That Cannot Die Halfway#

Most teams evaluate agents on answer quality. The metric that actually decides whether you can sell the feature is different: what happens when a run is interrupted?

Interruptions are not rare edge cases. A production agent run is a chain of slow, unreliable operations. LLM calls take seconds and get rate-limited. Tool calls hit third-party APIs that time out. Runs that involve research or multi-step reasoning can last minutes; runs that wait for a human approval can last days. Meanwhile your infrastructure keeps doing what infrastructure does: deploys restart containers, autoscalers kill pods, laptops close, serverless functions hit execution limits.

The business impact splits into three bills.

Reliability. A run that dies halfway either loses the customer's work or, worse, leaves it half-done. Half-done is the dangerous one. An agent that has already sent the email, written the database row, or charged the card cannot be safely re-run from the top. This is the "Day 2" crisis: the feature worked, customers adopted it, and now every infrastructure hiccup turns into a support ticket about duplicate invoices or vanished jobs.

Cost. When state lives only in process memory, the only recovery strategy is "run the whole thing again." Every retry re-pays for every LLM call that already succeeded. For a 20-step agent that fails at step 18, that is 17 successful calls thrown away and repurchased.

Latency and trust. Without persisted state you also cannot show progress, resume a session, or pause for approval. The customer sees a spinner, then an error, then nothing. Features like "the agent will ask you before sending" are impossible to build honestly if the agent cannot survive the wait.

Why the Naive Loop (and In-Prompt Patches) Fail#

The default agent shape, whether hand-rolled or generated by a framework quickstart, is an in-memory while-loop: call the model, execute the tool it picked, append the result to a message list, repeat until done. The message list is the entire state of the run, and it lives in RAM.

Teams usually try to patch this at the prompt layer first, because that is the layer they can see. The patches do not hold, because none of them change where state lives.

The pattern across every row: the failure happens below the model. Prompt engineering operates on what the model says; these failures are about what the infrastructure did and whether anyone wrote it down. That distinction is the whole argument for durable execution.

We compared the orchestration styles of LangGraph, CrewAI, and the OpenAI Agents SDK in our agent frameworks comparison, and dug into tool-level retry behavior in our tool failure recovery proof. Both of those operate at the framework layer. What follows is the layer underneath.

Production Architecture: Decompose the Loop into Durable Steps#

Durable execution runtimes solve the problem with one structural move: they split the agent into deterministic orchestration and retriable, checkpointed side effects, and they persist the boundary between them.

• In Temporal, the orchestration is a Workflow and each side effect (LLM call, tool call, database write) is an Activity. Every completed Activity is recorded in an append-only Event History. If the worker process crashes, another worker replays the history and resumes from the last completed step, with per-Activity retry policies handling transient failures. Temporal's AI cookbook documents the agent-specific patterns directly: Activity-backed tools, retry policies driven by HTTP responses, human-in-the-loop approval via Signals, and a Claim Check pattern that offloads large payloads to object storage so the history stays lean.
• In Inngest AgentKit (TypeScript, Apache-2.0, npm i @inngest/agent-kit inngest), agents compose into networks with a router deciding which agent acts next and shared network state carrying results between them. Runs execute on Inngest's step-based orchestration engine, which gives each step retry and fault-tolerance behavior in production, with human-in-the-loop and MCP tool support as documented patterns.

The blueprint, in Temporal's Python SDK. The workflow is the loop; every side effect is an Activity with an explicit retry policy:

from datetime import timedelta
from temporalio import workflow, activity
from temporalio.common import RetryPolicy

@activity.defn
async def call_llm(messages: list) -> dict:
    ...  # one model call; safe to retry, no side effects

@activity.defn
async def execute_tool(name: str, args: dict, idempotency_key: str) -> dict:
    ...  # one side effect, keyed so a retry cannot double-apply

@workflow.defn
class SupportAgent:
    @workflow.run
    async def run(self, task: str) -> str:
        messages = [{"role": "user", "content": task}]
        while True:
            decision = await workflow.execute_activity(
                call_llm, messages,
                start_to_close_timeout=timedelta(minutes=2),
                retry_policy=RetryPolicy(maximum_attempts=5),
            )
            if decision["type"] == "final_answer":
                return decision["content"]
            if decision["risk"] == "high":
                await workflow.wait_condition(lambda: self.approved)  # Signal sets this; can wait days
            result = await workflow.execute_activity(
                execute_tool, decision["tool"], decision["args"],
                f"{workflow.info().workflow_id}-{len(messages)}",
                start_to_close_timeout=timedelta(minutes=5),
                retry_policy=RetryPolicy(maximum_attempts=3),
            )
            messages.append(result)

Kill the worker at any point in that loop and the run resumes at the exact step it reached. The rate-limited LLM call retries by policy instead of crashing the run. The approval gate holds no process hostage.

The same shape in AgentKit, where the runtime draws the step boundaries for you:

import { createAgent, createNetwork, anthropic } from "@inngest/agent-kit";

const triage = createAgent({
  name: "triage",
  system: "Classify the request and pick a specialist.",
  model: anthropic({ model: "claude-sonnet-5" }),
});

const network = createNetwork({
  name: "support-network",
  agents: [triage, refundAgent, escalationAgent],
  router: ({ network }) =>
    network.state.data.classified ? undefined : triage,
});

Each agent inference and tool call runs as a durable step on Inngest's engine; network state, not process memory, carries results between agents, and the Dev Server traces every step locally before you deploy.

Three rules make either runtime work, and they are yours to enforce rather than the vendor's:

1. Side effects only in Activities/steps, keyed for idempotency. Recovery means retrying, and a retried charge must be a no-op. This is the same discipline as idempotent tool design, applied at the runtime layer.
2. Keep the orchestration deterministic. No direct network calls, clocks, or randomness in workflow code; replay depends on it.
3. Keep payloads out of the history. Big documents and tool outputs go to object storage with references in state, per Temporal's Claim Check pattern.

Can this survive your workflow?#

Before adopting, walk your highest-value agent run through four questions. If any answer is "no," you have found the incident that is coming.

• If the process dies after step N, does anything on disk know steps 1 through N happened?
• If the customer clicks "retry," can any completed side effect run twice?
• Can a run pause for a human decision overnight without holding a process open?
• When a tool call times out, can you tell "it never ran" from "it ran and the response was lost"?

The Financial Case for Founders#

Durable execution is infrastructure, so its ROI shows up as costs that stop happening. Three of them dominate, and you can size each from your own numbers rather than anyone's benchmark.

Wasted model spend on re-runs. With in-memory state, expected token cost per completed run is roughly (cost per attempt) × (average attempts to survive uninterrupted). Checkpointing changes the unit of retry from "the run" to "the step": a failure at step 18 of 20 re-pays one step, not seventeen. To size this, multiply your current failed-run rate by your average per-run model cost; that product is your monthly re-run bill. It grows linearly with run length, which means it grows exactly as your agents become more capable.

Incident cost from duplicate side effects. One duplicated refund or double-sent contract can consume more support and goodwill than a month of model spend. The idempotency-keyed Activity pattern converts this from an incident category into a non-event. This is the line item that matters most if your agent touches money, records, or customer communication.

Engineering time spent rebuilding the runtime. Teams that keep the naive loop end up hand-building persistence, retry ledgers, resume logic, and approval queues one incident at a time. That is months of senior engineering spent re-implementing what Temporal and Inngest ship, tested, on day one. The build-vs-adopt question is not whether you can build it; it is whether runtime engineering is the thing your customers are paying you for.

The honest offset: durable execution adds a per-step persistence hop, a new service (or vendor) to operate, and the determinism discipline above. For a stateless chat feature, that is overhead without payoff. For an agent that acts on the world, it is the difference between a demo and a product. Note also what durable execution does not fix: it retries and resumes, but it cannot make a bad decision good. Cost-per-decision is a separate lever, which is where hybrid model routing and token-level optimization pick up.

When to Use, When to Skip#

Use durable execution when:

• Agent runs perform side effects a customer would notice twice (payments, emails, writes to their systems)
• Runs exceed a few seconds or span multiple tool calls, deploys can interrupt them, and re-running from zero is expensive
• You need human-in-the-loop approval gates that hold for hours or days
• Runs must be auditable: a step-by-step history of what executed is a compliance asset

Skip it when:

• The feature is single-shot inference or read-only chat with no side effects; a plain API call with client-side retry is simpler and cheaper
• Your whole agent run reliably completes in seconds and losing one costs a shrug, not a ticket
• The team cannot yet hold the determinism boundary; a misused durable runtime that hides side effects inside workflow code fails in stranger ways than the loop it replaced

One limitation to state plainly: neither runtime has meaningful published head-to-head reliability numbers under identical agent workloads, so choosing between them is an architecture-and-team decision (TypeScript-native, managed, batteries-included networks with AgentKit; language-flexible, self-hostable, maximum-control Workflows with Temporal), not a benchmark decision.

What to Do Differently on Monday#

Inventory every agent feature you run and mark the ones with side effects. For each, answer the four survival questions above. If any fail, draw the workflow/activity boundary through your existing loop on paper first: which lines are orchestration, which are side effects, and what is each side effect's idempotency key. That one-page exercise is most of the migration; the runtime choice comes after.

What Effloow added: the failure-trigger table mapping each production interruption to the exact reason prompt-level fixes cannot address it, the four-question survival audit, and a founder-facing cost model you can populate from your own failed-run rate. All three are synthesized from the primary Inngest and Temporal documentation, not a restatement of either vendor's pitch.

If you are shipping an agentic feature and the four-question audit came back with a "no," this is exactly the class of architecture work we take on. See our services or contact us to talk through your agent's failure modes before your customers find them; our Proof Studio also produces evidence-bound write-ups of exactly how a given agent stack behaves under failure.

For Your Engineers#

Primary sources worth reading in full before choosing a runtime:

• Inngest AgentKit: agentkit.inngest.com/overview and the repo, TypeScript, @inngest/agent-kit (with inngest as a peer dependency from v0.9.0), Apache-2.0. Composable agents/networks/routers with shared state, MCP servers as tools, OpenAI-compatible plus Anthropic and Gemini model support, human-in-the-loop as a documented pattern, and step-level tracing through the Inngest Dev Server.
• Temporal for AI: docs.temporal.io/ai-cookbook and temporal.io/ai. The cookbook's recipes cover the durable agentic loop, Activity-backed tools for OpenAI, Claude, MCP, and the Vercel AI SDK, retry policies derived from HTTP responses, Signals for approval gates, post-LLM deterministic guardrails, and the Claim Check pattern for large payloads.

Migration notes from the blueprint above: keep every non-deterministic call out of workflow code (model calls included, since they are Activities); derive idempotency keys from workflow ID plus step position, not timestamps; set start_to_close_timeout on every Activity, because unbounded Activities are the durable-runtime equivalent of the hung while-loop; and budget for Event History size from day one if your tools return documents.