Stop Averaging p95: A Python Lab for API Latency Histograms

Why We Brought This Tool Into Our Lab#

A service dashboard can show a reassuring latency number while answering the wrong question. Suppose each worker reports its p95 and the dashboard averages those values. That calculation describes the average of worker statistics. It doesn't recover the latency below which most requests across the service finished. Weighting those statistics by request count sounds like a repair, but it still throws away the distribution you needed.

Our fixed Python fixture makes that distinction visible. Its pooled nearest-rank p95 is 900 milliseconds. Averaging the worker p95 values returns 495.0 milliseconds; weighting them by request count returns 163.636364 milliseconds. These are calculations over authored inputs, not measurements of an API. Nothing became faster between those answers. Only the aggregation changed.

The tool in this lab is a small, standard-library histogram harness. It preserves the observations long enough to calculate an exact reference, then deliberately discards detail by bucketing them. That gives us two separate questions: did merging preserve the bucket counts, and how closely does interpolation recover the reference percentile? A correct answer to the first does not guarantee an exact answer to the second.

Prometheus's histogram practices guide explains the underlying distinction: precomputed quantiles cannot generally be aggregated into a service quantile, whereas histogram distributions support aggregation. Its current guidance prefers native histograms where possible. We use explicit classic-style buckets here because every count and boundary remains easy to inspect, not because classic histograms are the default recommendation for new deployments.

This matters before you compare gateways, retrieval engines, or model servers. A latency comparison built on incompatible aggregation rules can send an optimization effort toward the wrong component. Tracing addresses a different question: which steps consumed time within a request. Our OpenTelemetry agent-tracing guide covers that diagnostic view; this lab addresses the distribution across requests.

The result is deliberately narrow. We exercised Python arithmetic, matching-bucket merges, boundary cases, and a layout rejection. We did not deploy Prometheus, export OpenTelemetry metrics, call a vendor API, or benchmark a serving system.

Hands-On Walkthrough: Setup, Execution and Output#

Use Python 3 with its standard library. Save the exact recipe below as recipe.py and invoke python3 -I -S recipe.py. No package installation, credentials, network endpoint, or external dataset is needed. Keep assertions enabled; running with optimization would remove checks that make the example useful.

All durations are milliseconds. The normal fixture represents disjoint workers observing one common window. The first has eighteen observations at 10 and two at 90. The second has two observations at 900. These deliberately uneven populations expose both the equal-worker average and the request-weighted average without relying on random data or timing variability.

The exact reference uses nearest rank: sort observations and select the item at the ceiling of the target fraction times the population size. This choice is explicit because percentile conventions differ. Here, “exact” means exact for that empirical definition and these inputs, not an estimate of some underlying production distribution.

The histogram function returns cumulative counts at each finite upper bound, followed by the total population for an implicit positive-infinity bucket. The merge function adds corresponding counts only when layouts match. The estimator then uses linear interpolation inside the bucket containing the target rank, assuming a zero lower bound for the first positive bucket.

import json
import math

# Milliseconds; disjoint workers observing the same fixed window.
# Exact p95 uses nearest rank, not a streaming-summary algorithm.
def exact(values):
    return sorted(values)[math.ceil(0.95 * len(values)) - 1]

def histogram(values, bounds):
    if not bounds or not all(type(b) in (int, float) and math.isfinite(b) and b > 0 for b in bounds):
        raise ValueError('positive finite bounds required')
    if any(a >= b for a, b in zip(bounds, bounds[1:])):
        raise ValueError('strictly increasing bounds required')
    if not values or not all(type(v) in (int, float) and math.isfinite(v) and v >= 0 for v in values):
        raise ValueError('nonempty finite nonnegative observations required')
    # Final count represents the implicit +Inf bucket.
    return [sum(v <= b for v in values) for b in bounds] + [len(values)]

def merge(left_bounds, left, right_bounds, right):
    if left_bounds != right_bounds or len(left) != len(right):
        raise ValueError('bucket layouts must match')
    return [a + b for a, b in zip(left, right)]

def estimate(bounds, counts):
    target = 0.95 * counts[-1]
    low, previous = 0, 0
    for high, count in zip(bounds, counts[:-1]):
        if count >= target:
            return low + (high - low) * (target - previous) / (count - previous)
        low, previous = high, count
    # Classic histogram rule when the rank falls in +Inf.
    return bounds[-1]

bounds = [20, 100, 500, 1000]
workers = [[10] * 18 + [90] * 2, [900] * 2]
pooled = workers[0] + workers[1]
local = [exact(w) for w in workers]
counts = [histogram(w, bounds) for w in workers]
merged = merge(bounds, counts[0], bounds, counts[1])
truth = exact(pooled)
unweighted = sum(local) / len(local)
weighted = sum(p * len(w) for p, w in zip(local, workers)) / len(pooled)
coarse = estimate(bounds, merged)
fine_bounds = [20, 100, 500, 800, 900, 1000]
fine = estimate(fine_bounds, histogram(pooled, fine_bounds))
assert merged == histogram(pooled, bounds)
assert unweighted != truth and weighted != truth
assert abs(fine - truth) < abs(coarse - truth)
print(json.dumps({'case': 'normal', 'unit': 'ms', 'workers': workers, 'bounds': bounds, 'worker_cumulative_counts': counts, 'merged_cumulative_counts': merged, 'worker_p95': local, 'mean_worker_p95': unweighted, 'count_weighted_worker_p95': round(weighted, 6), 'pooled_nearest_rank_p95': truth, 'merged_linear_p95': round(coarse, 6), 'fine_bounds': fine_bounds, 'fine_linear_p95': round(fine, 6)}))

edge = [100] * 19 + [900]
edge_counts = histogram(edge, bounds)
assert edge_counts[1] == 19
assert exact(edge) == 100 and estimate(bounds, edge_counts) == 100
overflow = [1500]
assert estimate(bounds, histogram(overflow, bounds)) == bounds[-1] < exact(overflow)
print(json.dumps({'case': 'boundary', 'bounds': bounds, 'edge_inputs': edge, 'cumulative_counts': edge_counts, 'exact_p95': exact(edge), 'linear_p95': estimate(bounds, edge_counts), 'overflow_inputs': overflow, 'overflow_exact_p95': exact(overflow), 'overflow_linear_p95': estimate(bounds, histogram(overflow, bounds))}))

wrong_bounds = [20, 100, 600, 1000]
rejected = False
try:
    merge(bounds, counts[0], wrong_bounds, histogram(workers[1], wrong_bounds))
except ValueError as error:
    rejected = True
    print(json.dumps({'case': 'invalid', 'left_bounds': bounds, 'right_bounds': wrong_bounds, 'error': str(error)}))
assert rejected

The recorded host execution on September 10, 2026 exited with code 0, produced no stderr, and recorded unchanged recipe inputs before and after execution. Its full stdout follows. These lines are the captured Python result, not illustrative output from a monitoring vendor.

{"case": "normal", "unit": "ms", "workers": [[10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 90, 90], [900, 900]], "bounds": [20, 100, 500, 1000], "worker_cumulative_counts": [[18, 20, 20, 20, 20], [0, 0, 0, 2, 2]], "merged_cumulative_counts": [18, 20, 20, 22, 22], "worker_p95": [90, 900], "mean_worker_p95": 495.0, "count_weighted_worker_p95": 163.636364, "pooled_nearest_rank_p95": 900, "merged_linear_p95": 725.0, "fine_bounds": [20, 100, 500, 800, 900, 1000], "fine_linear_p95": 845.0}
{"case": "boundary", "bounds": [20, 100, 500, 1000], "edge_inputs": [100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 900], "cumulative_counts": [0, 19, 19, 20, 20], "exact_p95": 100, "linear_p95": 100.0, "overflow_inputs": [1500], "overflow_exact_p95": 1500, "overflow_linear_p95": 1000}
{"case": "invalid", "left_bounds": [20, 100, 500, 1000], "right_bounds": [20, 100, 600, 1000], "error": "bucket layouts must match"}

Read the normal record in order. The individual p95 values are 90 and 900. Neither averaging method recovers the pooled reference. By contrast, merging worker histograms produces exactly the same cumulative counts as bucketing the pooled observations directly. That equality is asserted before the result is printed.

The merged estimate is still 725.0 rather than 900. Adding boundaries produces 845.0 on the same observations. Refinement helps this fixture, but it does not make interpolation exact or establish a universal accuracy improvement. The estimator knows bucket populations, not where the observations sit inside each bucket.

For easier inspection, paste one output line at a time into our JSON formatter. The output is newline-delimited JSON: each line is a complete object, while the entire block is not a single JSON document. Formatting changes readability, not the statistical meaning of its fields.

What Broke: The Gotchas and Limitations We Hit#

The first failure was conceptual: local quantiles had already discarded the information needed for pooling. Request weighting cannot restore that information. This is different from combining means using their population sizes. The relevant inputs for a pooled percentile are observations or an aggregable representation of their distribution, not just local percentile values.

The second failure was resolution. Our coarse histogram merges correctly and still estimates below the nearest-rank reference. The Prometheus query-function reference documents uniform-within-bucket interpolation for classic histograms and native histograms with custom boundaries. It separately documents exponential interpolation for nonzero buckets using standard exponential schemas. The Python estimator models the former rule only.

The boundary fixture checks two details together. Observations exactly at 100 belong in the bucket whose upper bound is 100, and the percentile rank lands on that cumulative boundary. Both reference and estimate return 100. Inclusive bounds matter: replacing the comparison with a strict inequality would change the counts and invalidate the intended test.

Overflow is more surprising. With a single observation at 1500, the reference is 1500 but the estimate returns 1000. That is the documented classic-histogram fallback when the quantile falls in the positive-infinity bucket: return the highest finite boundary. It does not establish that the request completed within that boundary. A dashboard must not mistake this fallback for a measured upper limit.

The layout check rejects a merge when one worker uses 500 and another uses 600 at the corresponding boundary. Equal array length is insufficient. This strict rejection is our lab policy, not a claim that every Prometheus histogram representation rejects differing layouts. The practices guide describes reconciliation for native representations, including possible loss of resolution.

A separate translation trap appears in the OpenTelemetry Metrics Data Model. Its explicit histogram buckets contain populations within individual intervals. Our arrays contain cumulative counts across upper bounds. Both can describe the same observations, but their arrays are not interchangeable. OpenTelemetry also defines cumulative and delta temporality, which concerns accumulation over time. That is independent of cumulative counting across bucket boundaries.

Finally, this estimator is not a replacement for histogram_quantile. It accepts internally generated nonempty counts, fixes the requested quantile, and omits malformed-count repair, counter resets, rate calculations, negative observations, and native exponential schemas. Empty traffic needs an explicit policy in a real application; this recipe rejects empty observation lists rather than quietly calling them zero latency.

Scale, Latency and Cost vs. Alternatives#

Choose the representation according to the question you need to answer. Keeping raw observations permits an exact empirical percentile under a declared convention, but also retains every input. The recipe intentionally keeps those values to provide a reference. That is convenient for a small regression fixture, not evidence that retaining all production request durations is economical.

Matching histograms retain less detail and can be merged without reconstructing individual requests. The price is uncertainty within buckets. More boundaries can make that uncertainty more useful around an operational threshold, but they also create more state to collect and process. The practices guide notes that classic exposition creates separate bucket series, making boundary selection a resource decision as well as an accuracy decision.

Native representation changes those storage mechanics. The Prometheus maintainer's February 2026 composite-types article explains the move from sets of primitive samples toward composite samples. It also warns that some discussed changes were not yet approved or proven in production. We use that source for architectural context, not as evidence of a measured storage discount or query-speed advantage.

For an existing classic Prometheus setup, the documented aggregation order is to calculate rates, sum matching bucket series while retaining the boundary label, and then calculate the quantile. Our fixed-window addition illustrates the distribution-preservation step. It does not test scraping, temporal alignment, resets, or the query engine. Those need separate verification against the deployed metric and labels.

Sometimes a percentile is not the most direct operational question. If you need the fraction of requests meeting a fixed latency target, a bucket placed exactly at that target can answer it without interpolating a percentile. The practices guide documents that approach and warns about missing boundaries across participating classic histograms. A threshold query still needs complete, compatible populations.

There is no latency or dollar winner from this run. The script uses no paid API, but its execution record is not a production cost model. Before buying more capacity, establish that the dashboard measures the intended request population. Then compare instrumentation overhead, retention requirements, and query behavior on the actual stack rather than turning this arithmetic fixture into a benchmark.

Our Final Verdict: When to Deploy, When to Skip#

Use this harness as a regression example when reviewing service-wide percentile calculations, changing histogram boundaries, or explaining why request-weighted worker p95 is not a pooled p95. Its strongest property is inspectability: every observation is printed, every boundary is explicit, and the expected failures are checked alongside the successful merge.

What Effloow added is the executable contrast between aggregation failure and interpolation error, plus boundary, overflow, and incompatible-layout controls. The normal result makes the distinction hard to miss: merging preserves the represented distribution while estimating a quantile can still lose precision. Those are separate acceptance checks and should remain separate in your own tests.

Before adopting a dashboard change, write down which requests count, the common observation window, the duration unit, and the percentile convention. Confirm whether bucket arrays are interval populations or cumulative counts. Decide what should happen when observations exceed the finite range. If those answers are unclear, a more polished graph will not fix the measurement.

Skip this implementation as a production quantile library or a shortcut to claiming Prometheus compatibility. Skip worker-percentile averaging whenever the intended output is a request-level pooled percentile. And skip a histogram migration undertaken solely because this fixture improved with extra boundaries: your latency distribution and supported telemetry path determine whether that trade is worthwhile.

The practical change is small: aggregate distributions before calculating service percentiles, label interpolated estimates honestly, and keep a fixed counterexample beside the dashboard logic. A trustworthy latency number starts with a defined population and a reproducible calculation.