Beyond pass@1: How Agent Reliability Decays in Long-Horizon Tasks

Most agent benchmarks report a single number: pass@1. Your agent either solved the task or it didn't. That's useful for comparing models on short, deterministic tasks. It falls apart the moment you care about real-world deployments, where tasks span dozens of steps and a single recoverable error early on can cascade into total failure by the end.

arXiv:2603.29231, "Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents," formalizes this problem. The paper evaluates 10 language models across 23,392 episodes and introduces a four-metric framework — RDC, VAF, GDS, and MOP — to measure not just whether an agent succeeds, but how it fails as task complexity grows. The headline finding: mean pass@1 falls from 76.3% on short tasks to 50.5% on long ones (arXiv:2603.29231), and the decay is not linear.

Effloow Lab ran a local Python simulation to make all four metrics concrete and observable. This is a conceptual simulation using synthetic trajectories — it does not reproduce the paper's 23,392-episode dataset. All simulation numbers below come from local execution; all paper percentages are sourced from arXiv:2603.29231.

Why pass@1 Alone Tells You Almost Nothing About Long-Horizon Agents

Imagine two agents, both with 80% pass@1 on a 5-step task. Now you deploy them on a 20-step task. Agent A drops to 65%. Agent B drops to 20%. Same headline metric on short tasks, wildly different behavior at scale.

This is the gap the paper addresses. A single pass@1 score conflates two distinct properties: raw capability (can the model reason?) and reliability (does that capability hold as the task grows?). When you're building production systems — automated code review pipelines, multi-step data extraction agents, document automation workflows — you need to know both.

The paper's framing: think of agent reliability the way reliability engineers think about physical systems. Not "did it work once?" but "what is the failure mode, where does it appear, and how does it degrade?" That shift in framing is what the four metrics operationalize.

The Four Metrics: What They Measure and Why They Matter

Before running the simulation, here is a quick reference for what each metric captures.

Think of them as four angles on the same underlying question: when you stretch a task out, what breaks, how fast, and how predictably?

RDC is the most direct — you plot pass@1 against task length buckets and read the curve. VAF tells you whether that curve is also becoming noisier (high variance means unpredictable behavior). GDS rewards agents that degrade smoothly rather than cliff-diving. MOP gives you a threshold you can actually use in system design: "beyond X steps, this agent is unreliable."

Paper Results: What 23,392 Episodes Revealed

The paper's empirical RDC across 10 models and 23,392 episodes (all figures from arXiv:2603.29231):

• Short tasks: mean pass@1 = 76.3%
• Medium tasks: mean pass@1 = 59.8%
• Long tasks: mean pass@1 = 50.5%
• Very long tasks: mean pass@1 = 52.1%

The very-long bucket shows a slight uptick over long — the paper discusses this in detail, attributing it partially to task selection effects at the far end of the length distribution. The core pattern is clear: reliability drops materially from short to long, and the drop accelerates in the middle range (short → medium is the steepest decline).

Crucially, the paper identifies super-linear decay as the dominant failure mechanism. This is not the naive "each step has 5% failure rate and they compound" model. Instead, errors in LLM agent trajectories tend to persist: once an agent makes a wrong tool call, misinterprets a context, or takes a wrong branch, subsequent steps are more likely to fail because they build on a corrupted state. The failure propagates. This is the error persistence effect.

Effloow Lab Simulation: Running All Four Metrics Locally

Effloow Lab implemented all four metrics in a self-contained Python stdlib simulation. The core mechanic is an in_error_state flag: when a step fails, the agent enters error state and subsequent steps have a 65% chance of staying in that state — modeling the paper's error persistence observation. Random seed is fixed at 42 for reproducibility.

This is a conceptual simulation only. No real LLM is involved. Results are not comparable to the paper's numbers. The purpose is to show how each metric is computed and what the values mean.

def simulate_agent_trajectory(n_steps, base_success_rate=0.88, error_correlation=0.65):
    steps = []
    in_error_state = False
    for i in range(n_steps):
        if in_error_state:
            success = random.random() > error_correlation
        else:
            success = random.random() < base_success_rate
        steps.append(success)
        in_error_state = not success if not success else (in_error_state and not success)
    return all(steps)

The error_correlation=0.65 parameter was chosen to make error persistence visible in the output — not to calibrate against paper data.

Simulation Output: The Numbers From Local Execution

Running python3 /tmp/effloow-reliability-poc.py (Python 3.12, macOS, random.seed(42)) produced:

Reliability Decay Curve (RDC):
  short     :  67.5%  █████████████
  medium    :  38.0%  ███████
  long      :  21.5%  ████
  very_long :   3.5%  

Variance Amplification Factor (VAF): 0.833
Graceful Degradation Score (GDS):    0.36
Meltdown Onset Point (MOP):          medium

The simulation's decay is steeper than the paper's because it uses a single synthetic agent with a fixed error correlation rather than averaging across 10 frontier LLMs. The structural pattern — accelerating decay with error persistence — is consistent with the paper's core finding.

What these numbers tell you:

• RDC: pass@1 collapses from 67.5% at 3 steps to 3.5% at 25 steps. The drop from short to medium (67.5% → 38.0%) is steeper than medium to long, consistent with the paper's observation about the middle-range acceleration.
• VAF of 0.833: High. The spread in outcomes across buckets is large relative to the mean. Practically this means: you cannot predict how this agent will behave on a new task just by knowing its short-task pass@1.
• GDS of 0.36: Low. A well-designed agent would stay close to 1.0. At 0.36, this agent cliff-dives rather than gracefully degrading — the kind of behavior that makes production deployments unreliable.
• MOP at medium (8 steps): The threshold drop of 15 percentage points from the short baseline fires at the medium bucket. In system design terms: beyond 8 steps, plan for fallbacks.

Super-Linear Decay: Why Errors Compound Faster Than You Expect

The naive model of agent failure goes like this: if each step has a 10% failure rate, a 10-step task succeeds with probability 0.9^10 = 34.9%. That's already bad. But the paper shows real agents fail faster than this independent-steps model predicts.

The mechanism is error persistence. In real LLM agent trajectories, a failed step typically leaves the agent in a corrupted state: the context window now contains a wrong tool result, a misidentified file, or an incorrect intermediate value. The next step doesn't start from a clean slate — it starts from a broken one. So the probability of step N failing is not independent of step N-1; it is conditional on it. Once the agent is off track, subsequent steps are disproportionately likely to stay off track.

The simulation models this directly. With error_correlation=0.65, an agent in error state has only a 35% chance of recovering each step, rather than an 88% base success rate. That asymmetry is what drives the steep drop from 67.5% to 3.5% across the length buckets.

In the paper's empirical data, this manifests as a decay curve that is steeper in the middle range than a purely multiplicative model would predict. Frontier models soften this because they have stronger context understanding and recovery strategies — but even they don't escape it entirely, as the 76.3% → 50.5% drop across the paper's model pool shows (arXiv:2603.29231).

Practical Guide: Evaluating Your Own Agent with These Metrics

You don't need the paper's 23,392-episode infrastructure to apply this framework. Here's how to run a lightweight version on your own agent.

Step 1: Define task length buckets. Pick four ranges that make sense for your domain. For a code-review agent: 1-3 files (short), 4-10 files (medium), 11-20 files (long), 21+ files (very long). For a data extraction agent: row counts, document pages, or pipeline stages.

Step 2: Sample 50-100 tasks per bucket. Run your agent, record binary pass/fail. This is your empirical RDC. Plot pass@1 against bucket. If the curve is flat, your agent is robust. If it drops sharply, identify the bucket where the drop begins.

Step 3: Compute MOP. Find the first bucket where pass@1 drops more than 15 percentage points from your short-task baseline. That's your deployment ceiling for fully autonomous operation.

Step 4: Compute GDS. GDS = 1.0 - (max_pass - min_pass) / 100. Values below 0.5 mean your agent is brittle. Values above 0.8 mean it degrades smoothly enough to be trustworthy at scale.

Step 5: Compute VAF. VAF = stdev(pass_rates) / mean(pass_rates). High VAF (above 0.5) means unpredictable behavior — the same agent might score 70% one day and 30% another on tasks of the same length bucket. Low VAF means reliable, predictable performance.

Step 6: Use MOP to set checkpoints. For tasks that exceed your MOP, insert human review or automated verification at the MOP boundary. Don't let the agent run unchecked past the threshold where your data shows reliability collapse.

This process takes one afternoon with a small task sample. It gives you far more actionable information than a single pass@1 number.

How This Connects to Other Reliability Research

This framework doesn't exist in isolation. Two related threads worth knowing:

Tool use reliability. The WildToolBench benchmark evaluates agents on realistic, multi-turn tool use scenarios. The core challenge it surfaces — agents failing when tool schemas diverge from training distribution — is a specific instance of the error persistence problem. A wrong tool call in step 3 propagates into a corrupted state by step 10. RDC and MOP would capture exactly this failure mode if WildToolBench tasks were profiled by length.

Autonomous development agents. The meta-agent challenge paper documents how autonomous coding agents fail on multi-file, multi-session tasks. The reliability decay framework applies directly: those agents would likely show a MOP somewhere in the 5-10 step range for complex refactoring tasks, and their GDS would be low (they tend to succeed completely or fail completely, not degrade gracefully).

The common thread is that current benchmarks optimize for short-horizon, single-shot evaluation. The paper argues — persuasively — that this creates a systematic blind spot when deploying agents on real work.

Simulation vs. Paper: What the Numbers Actually Show

It's worth being explicit about what the local simulation does and doesn't demonstrate.

The paper's agents maintain ~50% reliability on very long tasks because frontier LLMs have sophisticated error recovery. The simulation collapses to 3.5% because the synthetic agent has no recovery strategy beyond the base step success rate. Neither number is "right" or "wrong" — they reflect different systems. The simulation's value is making the metric computation transparent, not replicating the paper's results.

FAQ

Does the simulation reproduce the paper's results? No. The local simulation uses a synthetic single-agent model with 200 trials per bucket. The paper evaluates 10 real LLMs across 23,392 episodes. The structural pattern (reliability decays with task length) is consistent, but the specific numbers are not comparable. All paper percentages in this article are sourced from arXiv:2603.29231.

What is error persistence and why does it cause super-linear decay? Error persistence means that once an agent makes a mistake, subsequent steps are more likely to fail because they build on a corrupted intermediate state. This is different from the naive independent-steps failure model. The simulation models it with an in_error_state flag and a 65% probability of staying in that state after a failure. The result is decay that accelerates faster than a simple per-step failure rate would predict.

Can I compute these metrics without running 23,000 episodes? Yes. As shown in the practical guide section, 50-100 tasks per bucket is enough to get directionally correct RDC, GDS, VAF, and MOP values. The numbers will have more variance, but the structural shape of the curve — and the approximate MOP location — will be informative enough for system design decisions.

What is a "good" GDS value? The paper frames GDS as a robustness measure where 1.0 is perfect (no degradation at all) and 0.0 is complete collapse. In practice, values above 0.7 indicate that performance falls gradually rather than abruptly, which is what you want in a deployed system. Values below 0.5 — like the 0.36 in the simulation — indicate the agent tends to succeed or fail completely rather than degrade gracefully.

How does VAF differ from just reporting standard deviation? Standard deviation alone doesn't account for the mean level of performance. An agent with 70% ± 10% pass@1 is very different from one with 20% ± 10% — the latter is barely working at all, so the variance barely matters. VAF normalizes standard deviation by the mean, making it a coefficient of variation that is meaningful across different performance levels and task lengths.

All simulation code is available in the Effloow Lab run file. All paper figures are sourced from arXiv:2603.29231 and attributed throughout.