Why is comparing average pass rates not enough to approve a change?

A change can break one category completely while improving others, leaving the average almost unchanged; per-case and per-type comparisons reveal it.

How do you tell whether a pass-rate difference is real?

Compute confidence intervals for each pass rate (for example Wilson intervals), and for noisy outputs run cases several times; small differences within overlapping intervals may be noise.

What should an eval gate in CI block?

Regressions in critical case types (for example refunds, safety), and any newly failing case that has not been reviewed; small average drops elsewhere can be warnings.

Evals in CI: Catching Regressions Before Users Do — LLM Evals and Guardrails

Scenario

Example company: the team now has a 200-case eval set and runs it before releases. A prompt change drops the pass rate from 86 % to 83 %; another change raises it from 83 % to 85 %. Each time the team argues about whether the change is real.

What you see in production

A change that lost "only" 1.5 points shipped. It broke every Spanish refund question (7 cases) while slightly improving others, so the average barely moved. Users noticed within a day.

The naive code

before, after = run_evals(old_prompt), run_evals(new_prompt)
if after.pass_rate >= before.pass_rate - 0.05:      # "close enough"
    deploy(new_prompt)

Reproduce it

Confidence intervals for pass rates, and a per-case and per-type comparison that finds the hidden regression:

Pass rates with confidence intervals, and regressions by case typeTry it yourself

import math
from collections import Counter

def wilson(passes, n, z=1.96):
    p = passes / n
    centre = (p + z * z / (2 * n)) / (1 + z * z / n)
    margin = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / (1 + z * z / n)
    return centre - margin, centre + margin

types = ["booking"] * 80 + ["refund-en"] * 60 + ["refund-es"] * 7 + ["visa"] * 53
old = [True] * 172 + [False] * 28                      # 86 %
new = old.copy()
for i in range(140, 147):                               # all 7 Spanish refund cases now fail
    new[i] = False
for i in range(172, 176):                               # 4 old failures now pass
    new[i] = True

for name, results in [("old", old), ("new", new)]:
    lo, hi = wilson(sum(results), len(results))
    print(f"{name}: pass rate {sum(results) / len(results):.1%}  (95% CI {lo:.1%} to {hi:.1%})")
lost = Counter(t for t, o, n in zip(types, old, new) if o and not n)
won = Counter(t for t, o, n in zip(types, old, new) if not o and n)
print("newly failing by type:", dict(lost), "| newly passing by type:", dict(won))

import math
from collections import Counter

def wilson(passes, n, z=1.96):
    p = passes / n
    centre = (p + z * z / (2 * n)) / (1 + z * z / n)
    margin = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / (1 + z * z / n)
    return centre - margin, centre + margin

types = ["booking"] * 80 + ["refund-en"] * 60 + ["refund-es"] * 7 + ["visa"] * 53
old = [True] * 172 + [False] * 28                      # 86 %
new = old.copy()
for i in range(140, 147):                               # all 7 Spanish refund cases now fail
    new[i] = False
for i in range(172, 176):                               # 4 old failures now pass
    new[i] = True

for name, results in [("old", old), ("new", new)]:
    lo, hi = wilson(sum(results), len(results))
    print(f"{name}: pass rate {sum(results) / len(results):.1%}  (95% CI {lo:.1%} to {hi:.1%})")
lost = Counter(t for t, o, n in zip(types, old, new) if o and not n)
won = Counter(t for t, o, n in zip(types, old, new) if not o and n)
print("newly failing by type:", dict(lost), "| newly passing by type:", dict(won))

Output

old: pass rate 86.0%  (95% CI 80.5% to 90.1%)
new: pass rate 84.5%  (95% CI 78.8% to 88.9%)
newly failing by type: {'refund-es': 7} | newly passing by type: {'visa': 4}

Root cause

An average over 200 cases hides structure: a change can break one small category completely while improving others. And with a few hundred cases, a difference of a few points can be noise, especially with model graders or non-deterministic outputs. Thresholds on the average alone are both too loose and too strict.

The fix

Compare per case: list cases that newly fail and newly pass; review every newly failing case.
Break results down by type (intent, language, risk) and set stricter rules for high-risk types (for example, no regressions allowed in refunds).
Quantify uncertainty: report confidence intervals; for noisy outputs, run each case several times and use the pass fraction.
Gate in CI: run the eval suite on prompt, model, retrieval and tool changes; block on regressions in critical types, warn on small average drops.
Store results per version so you can compare any two versions later.

Trade-offs

Running large eval suites on every commit is slow and costly; run a fast subset on every change and the full suite before release.
Repeated runs per case multiply cost; reserve them for cases that are known to be noisy.
Strict gates block useful changes when the eval set itself is wrong; allow overrides with a reviewed justification and fix the cases.

Detect it in production

Eval results dashboard per version and per type, with the list of newly failing cases.
Production metrics by the same types, to confirm offline results match online behaviour.

Key points

Averages hide regressions; compare per case and per type.
Report confidence intervals and repeat noisy cases.
Gate changes in CI with strict rules for critical case types.

Further reading: Binomial proportion confidence interval (Wilson score) · Create strong empirical evaluations · Demystifying evals for AI agents