Sign in Get started

DevOps Tutorial · Operating in Production · Lesson 23 of 25

SLOs, Error Budgets and Alerts

Define service level objectives, turn them into error budgets, and alert on burn rate instead of every blip.

Create a free account to save this page, take notes and track your progress. Sign in

SLIs, SLOs and SLAs

  • An SLI (indicator) measures user experience: the share of requests that succeed within 300 ms.
  • An SLO (objective) is the target: 99.9% over 30 days.
  • An SLA is a contract with customers, with penalties; keep it looser than your SLO.

Error budgets

A 99.9% SLO allows 0.1% failures: the error budget. While budget remains, ship freely; when it's spent, slow down and invest in reliability.

Error budget per SLOTry it yourself
minutes = 30 * 24 * 60
for slo in [0.99, 0.999, 0.9999]:
    budget = minutes * (1 - slo)
    print(f"SLO {slo:.2%}: {budget:7.1f} minutes of full outage allowed per 30 days")

Output

SLO 99.00%:   432.0 minutes of full outage allowed per 30 days
SLO 99.90%:    43.2 minutes of full outage allowed per 30 days
SLO 99.99%:     4.3 minutes of full outage allowed per 30 days

Alert on burn rate

Page a human when the budget is being consumed fast (for example 14.4 times the sustainable rate over an hour, which spends 2% of a 30-day budget in that hour), and open a ticket for slow burns. Alerts should be about user-visible symptoms, not every CPU spike.

Burn-rate alertingTry it yourself
SLO = 0.999
budget_rate = 1 - SLO

def burn_rate(errors, requests):
    return (errors / requests) / budget_rate

windows = {"last 5 min": (90, 6_000), "last 1 hour": (1_100, 70_000), "last 6 hours": (1_300, 420_000)}
for name, (errors, requests) in windows.items():
    rate = burn_rate(errors, requests)
    action = "PAGE" if rate >= 14.4 else "ticket" if rate >= 1 else "ok"
    print(f"{name:12} error rate {errors / requests:.3%}  burn rate {rate:5.1f}x -> {action}")

Output

last 5 min   error rate 1.500%  burn rate  15.0x -> PAGE
last 1 hour  error rate 1.571%  burn rate  15.7x -> PAGE
last 6 hours error rate 0.310%  burn rate   3.1x -> ticket

Test yourself

A service has a 99.9% availability SLO over 30 days. About how much full downtime does its error budget allow?

  1. About 4 minutes
  2. About 43 minutes
  3. About 7 hours
  4. About 3 days
Show answer

Answer: About 43 minutes

0.1% of 43,200 minutes is 43.2 minutes.

Was this page helpful?