Free course · 11 lessons · Advanced
DevOps in Production
The DevOps failures that hurt real teams: rollbacks blocked by destructive migrations, secrets left in image layers, floating tags, OOMKilled and CPU-throttled containers, requests dropped on every deploy, full disks, alert fatigue and drifting environments, with runnable simulations and two capstone case studies.
1. Releases You Can Undo
A deployment is only safe if you can roll it back and rebuild it exactly. Migrations, secrets and floating tags break both.
- The Rollback That Made It Worse — A destructive schema migration ships with the code that needs it, so rolling back the code breaks production. Expand, migrate, then contract.
- The Key in the Image — Deleting a secret in a later Dockerfile step doesn't remove it from earlier layers. Use build secrets, .dockerignore and scanning, and rotate anything exposed.
- Same Commit, Different Build — Floating tags like latest or node:22 change under you. Pin base images by digest, build once, deploy by digest, and update dependencies on purpose.
2. Containers Under Load
Memory limits, CPU quotas and shutdown signals behave differently in containers. Size them from data and shut down gracefully.
- Restart Loop at Peak Traffic — Runtimes that don't respect the container's memory limit get OOMKilled under load. Size heap to the limit, set requests from data, and alert on restarts.
- Low CPU, High Latency — CPU limits are enforced per 100 ms period. Bursty services get throttled and p99 latency jumps even when average CPU looks low.
- Every Deploy Drops Requests — Pods that ignore SIGTERM or stop before load balancers notice drop in-flight requests. Exec-form CMD, readiness, a preStop delay and graceful shutdown fix it.
3. Running It Day to Day
The slow problems that eventually page someone at 3 a.m.: full disks, noisy alerts and environments that quietly drift apart.
- No Space Left on Device — Logs and images fill disks slowly, then everything fails at once. Log to stdout, rotate, garbage-collect images, and alert well before 100%.
- 300 Pages a Week — Alerts on every cause train people to ignore pages. Page on user-facing symptoms with SLO burn-rate alerts, and route the rest to tickets.
- It Worked in Staging — Staging that differs from production in config, versions and data hides bugs until release. Promote one artifact, keep config as code, and diff environments.
4. Capstone Case Studies
Design a delivery platform for twelve teams, and investigate a Friday deploy that took down checkout.
- Capstone: A Delivery Platform for Twelve Teams — Twelve product teams deploy monthly with hand-run scripts. Design the pipelines, runtime and practices that let them deploy daily and safely.
- Capstone: Postmortem — The Friday Deploy — A routine deploy took checkout down for 47 minutes. Reconstruct what happened from the evidence and write the remediation plan.