Online Evaluation and Monitoring

Scenario

Example company: a tax-help assistant passed its offline evals at launch in January. In April, during tax season, the question mix changes completely and a tax rule changes mid-month.

What you see in production

Nothing alerts. Error rates and latency are normal. Three weeks later, an accountant notices the assistant still quotes last year's deduction limit. Looking back, the share of answers mentioning outdated figures had been climbing since the rule change.

The naive code

# monitoring = uptime, error rate, latency
alert_if(error_rate > 0.01 or p95_latency_ms > 4000)

Reproduce it

Grade a daily sample of production answers and alert when the pass rate falls below its recent baseline:

import random
random.seed(2)

def daily_pass_rate(day):
    true_rate = 0.90 if day < 15 else 0.90 - 0.03 * (day - 14)      # a rule changes on day 15
    sample = [random.random() < true_rate for _ in range(60)]        # 60 graded answers per day
    return sum(sample) / len(sample)

history = []
for day in range(1, 26):
    rate = daily_pass_rate(day)
    baseline = sum(history[-7:]) / 7 if len(history) >= 7 else None
    alert = baseline is not None and rate < baseline - 0.12
    history.append(rate)
    if alert or day in (5, 14):
        print(f"day {day:2}: pass {rate:.0%}" + (f"  baseline {baseline:.0%}  ALERT" if alert else ""))

day  5: pass 88%
day 14: pass 88%
day 19: pass 72%  baseline 85%  ALERT
day 20: pass 70%  baseline 82%  ALERT
day 23: pass 48%  baseline 76%  ALERT
day 24: pass 57%  baseline 71%  ALERT
day 25: pass 50%  baseline 68%  ALERT

Root cause

Infrastructure metrics measure whether the system responds, not whether the answers are right. LLM quality drifts as traffic, content and the world change, and offline evals only cover what existed when they were written. Without measuring answer quality on live traffic, regressions surface through users.

The fix

• Sample live traffic daily (stratified by type), and grade it asynchronously with the same code and model graders as offline evals.
• Collect user signals: thumbs up/down, rephrasing, escalation to humans, abandoned conversations.
• Track metrics per type against a rolling baseline, and alert on significant drops.
• Watch the input distribution: new topics or languages appearing in traffic mean the eval set needs new cases.
• Route failures into the eval set, so offline evals learn from production.

Trade-offs

• Grading production samples costs money and needs privacy controls; sample, redact, and limit retention.
• Daily samples are noisy; alert thresholds must balance speed against false alarms (larger samples or multi-day windows).
• User feedback is sparse and biased toward unhappy users; use it as a signal, not a score.

Detect it in production

• Dashboards: graded pass rate by type, user feedback rate, escalation rate, refusal rate.
• Alerts on drops relative to baseline, and on shifts in the topic mix.

Key points

• Uptime and latency do not measure answer quality.
• Grade a sample of live traffic continuously and compare with a baseline.
• Feed production failures and new topics back into the eval set.

Further reading: Create strong empirical evaluations · Demystifying evals for AI agents