What's the difference between symptom-based and cause-based alerting?

Symptom-based alerts fire when users are affected (errors, latency against SLOs). Cause-based alerts fire on internal conditions (CPU, restarts) that may or may not hurt users; they belong on dashboards or tickets.

What is a burn-rate alert?

An alert on how fast the error budget is being consumed relative to the sustainable rate. Multi-window rules (e.g. 14.4x over 1 h and 5 min) page quickly for severe problems while ignoring brief blips.

How do you reduce alert fatigue on an existing system?

Measure pages per shift and actionability, demote non-actionable alerts, replace cause alerts with SLO burn-rate alerts, require runbooks, and review the top offenders every week.

300 Pages a Week — DevOps in Production

Scenario

Example team: a platform team has about 450 alert rules: CPU above 80%, memory above 75%, any pod restart, queue length above 100, disk above 70%, each copied per service. Everything pages the on-call engineer.

What you see in production

On-call receives 300 pages a week, most of which resolve on their own. Engineers mute channels and acknowledge without looking. When a real outage happens (checkout failing for 20% of users), its alert is lost among 15 others firing at the same time, and the first report comes from customer support 40 minutes later. Two engineers leave the rotation.

The naive code

- alert: HighCPU
  expr: avg(rate(container_cpu_usage_seconds_total[5m])) by (pod) > 0.8
  for: 1m
  labels: { severity: page }
- alert: PodRestarted
  expr: increase(kube_pod_container_status_restarts_total[5m]) > 0
  labels: { severity: page }

Reproduce it

Cause-based alerts fire on things that usually don't hurt users; symptom-based burn-rate alerts fire when users are actually affected:

Cause alerts versus SLO burn-rate alerts over a weekTry it yourself

import random

random.seed(8)
HOURS = 7 * 24
SLO_ERROR_RATE = 0.001                   # 99.9% success target

cause_pages = burn_pages = missed = 0
for hour in range(HOURS):
    cpu_spike = random.random() < 0.9    # some pod is above 80% CPU in most hours
    restarts = random.random() < 0.6
    outage = hour in (100, 101)          # a real incident: 3% of requests failing for two hours
    error_rate = 0.03 if outage else random.uniform(0, 0.0008)

    cause_pages += cpu_spike + restarts + outage    # outage also trips one cause alert
    burn_rate = error_rate / SLO_ERROR_RATE
    if burn_rate >= 14.4:
        burn_pages += 1
    elif outage:
        missed += 1

print(f"cause-based alerts: {cause_pages} pages in a week (2 of them mattered)")
print(f"burn-rate alerts:   {burn_pages} pages in a week, missed outages: {missed}")

Output

cause-based alerts: 243 pages in a week (2 of them mattered)
burn-rate alerts:   2 pages in a week, missed outages: 0

Root cause

Alerts were written for causes (CPU, restarts) instead of symptoms users feel (errors, latency). Most causes don't affect users (Kubernetes restarts pods; CPU at 85% is fine), so pages are mostly noise, and noise trains people to ignore pages. No one owned the alert set, so rules only ever got added.

The fix

Page only on symptoms tied to SLOs: availability and latency SLIs per user-facing service, with multi-window burn-rate alerts (for example 14.4× over 1 h and 5 min for paging, 6× over 6 h, and slower burns as tickets).
Demote cause alerts to dashboards or tickets: they help diagnose, but shouldn't wake anyone.
Every page must be actionable and link to a runbook; delete or fix any alert that fired without action.
Review alerts weekly in the on-call handover: count pages, classify (actionable, noise, duplicate), and fix the top offenders.

- alert: CheckoutErrorBudgetBurnFast
  expr: |
    (sum(rate(http_requests_total{service="checkout",code=~"5.."}[1h])) / sum(rate(http_requests_total{service="checkout"}[1h]))) > (14.4 * 0.001)
    and
    (sum(rate(http_requests_total{service="checkout",code=~"5.."}[5m])) / sum(rate(http_requests_total{service="checkout"}[5m]))) > (14.4 * 0.001)
  labels: { severity: page }
  annotations: { runbook: "https://runbooks.example.com/checkout-errors" }

Trade-offs

SLO alerts need good SLIs (request metrics at the edge) and agreement on targets with product owners.
Some causes deserve pages because they predict imminent outages (disk full within hours, certificate expiring tomorrow); keep these few and specific.
Fewer pages can feel risky at first; track missed incidents to prove coverage.

Detect it in production

Pages per on-call shift and the share that were actionable; target a handful per week.
Time from impact start to first page versus first customer report.
Alerts that auto-resolve within minutes, repeatedly: candidates for deletion.

Key points

Page on user-facing symptoms (SLO burn rate), not on every cause.
Every page must be actionable with a runbook; everything else is a ticket or a dashboard.
Review and prune alerts continuously; on-call health is a reliability metric.