Sign in Get started

DevOps Tutorial · Operating in Production · Lesson 24 of 25

Incident Response and Postmortems

Handle production incidents calmly: roles, mitigation before root cause, communication, and blameless postmortems.

Create a free account to save this page, take notes and track your progress. Sign in

During an incident

  1. Declare it early and open a channel; it's cheap to downgrade later.
  2. Assign roles: an incident commander coordinates, others investigate, one person communicates.
  3. Mitigate first: roll back the last deploy, turn off a feature flag, fail over, scale up. Find the root cause later; customers care about recovery.
  4. Communicate on a schedule (status page, stakeholders), even if there's no news.
  5. Keep a timeline of what was seen and done.
An incident timelineTry it yourself
from datetime import datetime

timeline = [("14:02", "error rate alert fires"), ("14:05", "incident declared, IC assigned"),
            ("14:11", "linked to 13:58 deploy"), ("14:13", "rollback started"),
            ("14:19", "error rate back to normal"), ("14:40", "incident closed")]
t = {event: datetime.strptime(ts, "%H:%M") for ts, event in timeline}
detect = (t["error rate alert fires"] - datetime.strptime("13:58", "%H:%M")).seconds // 60
mitigate = (t["error rate back to normal"] - t["error rate alert fires"]).seconds // 60
print(f"time to detect:   {detect} min (from the deploy)")
print(f"time to mitigate: {mitigate} min (from the alert)")
for ts, event in timeline:
    print(f"{ts}  {event}")

Output

time to detect:   4 min (from the deploy)
time to mitigate: 17 min (from the alert)
14:02  error rate alert fires
14:05  incident declared, IC assigned
14:11  linked to 13:58 deploy
14:13  rollback started
14:19  error rate back to normal
14:40  incident closed

After the incident: a blameless postmortem

Write down the impact, timeline, root cause and contributing factors, what went well, and action items with owners. "Blameless" means asking why the system allowed a mistake, not who made it, so people share what really happened.

Runbooks and on-call

  • Every alert links to a runbook: what it means, how to check, how to mitigate.
  • On-call rotations need sustainable load, handover notes, and time to fix the causes of pages.

Test yourself

A deploy 10 minutes ago correlates with rising errors. What should usually happen first?

  1. Find the exact bug in the code
  2. Roll back the deploy to restore service
  3. Write the postmortem
  4. Wait to see if it recovers
Show answer

Answer: Roll back the deploy to restore service

Mitigation comes before root cause; rolling back is usually the fastest safe mitigation.

Was this page helpful?