DevOps in Production · Capstone Case Studies

Capstone: Postmortem — The Friday Deploy

A routine deploy took checkout down for 47 minutes. Reconstruct what happened from the evidence and write the remediation plan.

Create a free account to save this page, take notes and track your progress. Sign in

Scenario

Example company: an online retailer. On Friday at 16:10, release 318 of the checkout service went out. Checkout errors started at 16:11; customers reported failures on social media at 16:20; the on-call engineer was paged at 16:24; service was restored at 16:58. About 9,000 orders failed.

Evidence

Release 318 included a migration: ALTER TABLE orders DROP COLUMN legacy_discount; (the new code no longer used it).
During the rolling update, old pods (still reading legacy_discount) failed every query on orders once the migration ran.
The first rollback attempt at 16:30 restored release 317, which failed the same way; the team then had to re-add the column by hand.
The checkout Dockerfile uses CMD sh -c "java $JAVA_OPTS -jar app.jar"; pod terminations took 30 s each.
Alerting: 140 alerts fired between 16:11 and 16:24 (CPU, restarts, queue length); the SLO alert existed but routed to a Slack channel, not the pager.
Friday afternoon deploys were "allowed but discouraged"; the reviewer approved the migration as "cleanup".

Deliverables

Root cause and the contributing factors.
Why did the rollback fail?
What should change, immediately and structurally?
How would you verify the changes work?

Draft each deliverable before revealing the model answer and rubric.

Further reading: ParallelChange (expand and contract) · Kubernetes pod termination

Deliverables

Root cause and contributing factors.

Model answer

Root cause: a destructive migration (dropping a column the previous release still read) ran while old pods were serving, breaking every old pod's queries.

Contributing factors: no migration linting or rule against destructive changes in the same release; the reviewer treated a schema drop as cleanup; shell-form CMD so pods ignored SIGTERM and took 30 s each, slowing the rollout and the recovery; the SLO alert wasn't routed to the pager while 140 noisy alerts were; detection came from customers before the page.

Rubric

Full marks: identifies the incompatible migration as the root cause and at least three contributing factors across review, runtime and alerting. Blameless framing.

Why did the rollback fail?

Model answer

Rolling back the code doesn't roll back the schema. Release 317 still read legacy_discount, which no longer existed, so it failed exactly like the old pods during the rollout. Recovery required a forward fix (re-adding the column).

Rubric

Full marks: explains code/schema asymmetry and why only a forward fix worked.

What should change, immediately and structurally?

Model answer

Immediately: route SLO burn-rate alerts to the pager and demote the cause alerts; exec-form CMD (CMD ["java", "-jar", "app.jar"] with JAVA_TOOL_OPTIONS) and SIGTERM handling with a preStop delay; a CI check that blocks DROP/RENAME in migrations without a contract label.

Structurally: expand/contract as the standard for schema changes (contract only after N releases); automated rollback testing in staging (deploy, migrate, roll back, smoke test); canary releases for checkout with automated analysis; alert review in the on-call handover.

Rubric

Full marks: fixes detection, migration safety, shutdown behaviour and release strategy. Banning Friday deploys alone isn't a fix.

How would you verify the changes work?

Model answer

Replay the scenario in staging: a migration that drops a used column must be blocked in CI; with an override, the rollback test must fail loudly before production.
Game day: deploy checkout during load testing and check zero failed requests across the rollout (graceful shutdown).
Inject errors in staging and confirm the burn-rate alert pages within minutes.
Track change failure rate and time to restore for checkout over the next quarter.

Rubric

Full marks: tests each control against the actual failure mode and defines ongoing metrics.

Open in the app to mark lessons complete, take notes and track your progress.