Free course · 13 lessons · Advanced
Kubernetes in Production
Run Kubernetes workloads through the failures teams actually hit: errors during deploys, OOMKills and evictions, dropped connections on shutdown, drains that take services down, config changes that never roll out, over-privileged service accounts, privileged pods, network policies that break DNS, autoscaling that flaps, upgrades blocked by removed APIs, CPU throttling and idle capacity. Examples run with kubectl against a real cluster or as Python simulations.
1. Deploying Without Breaking Things
The failures that hit almost every team's first Kubernetes rollouts: traffic to unready pods, wrong resource settings, dropped connections on shutdown and drains that take everything down.
- 502s During Every Deploy: Readiness Probes — Pods receive traffic before the app has started; readiness probes and rollout settings make deploys invisible to users.
- OOMKills, Evictions and Noisy Neighbours — Missing or wrong requests and limits cause OOMKilled pods, evictions and one service starving others; set them from measurements.
- Dropped Requests When Pods Shut Down — Pods are removed from load balancers after they receive SIGTERM, not before; handle the race with preStop delays and graceful shutdown.
- Node Drains That Take a Service Down — Cluster upgrades evict pods; without PodDisruptionBudgets and spread constraints, all replicas can disappear at once.
2. Configuration and Security
Config changes that never reach pods, over-privileged service accounts, privileged containers and network policies that break DNS.
- ConfigMap Changes That Never Reach the Pods — Updating a ConfigMap doesn't restart pods; tie configuration to rollouts with checksums or generated names.
- Over-Privileged Service Accounts — A compromised pod with broad RBAC permissions owns the cluster; audit what each identity can do and grant only what it needs.
- Privileged Containers and Pod Security Admission — A privileged pod or a hostPath mount is a path to the node; enforce the restricted Pod Security Standard per namespace.
- Network Policies That Break DNS — Locking down egress without allowing DNS breaks every service lookup; write default-deny policies that keep the cluster working.
3. Scaling, Upgrades and Cost
Autoscaling that flaps or never scales, upgrades blocked by removed APIs, CPU throttling from limits, and paying for idle capacity.
- Autoscaling That Flaps or Never Scales — The HPA scales on utilisation relative to requests; wrong requests and missing behaviour settings make it useless.
- Cluster Upgrades Blocked by Removed APIs — Kubernetes removes deprecated API versions; manifests and charts that still use them fail after an upgrade.
- Latency Spikes from CPU Limits — CPU limits throttle containers in 100 ms periods; bursty services suffer tail latency even when average usage looks low.
- Paying for Idle Capacity: Right-Sizing — Requests reserve capacity whether it's used or not; measure usage, right-size requests and let the cluster shrink.
4. Capstone
Review a workload for production readiness.
- Capstone: Production Readiness Review for a Payments API — Review and fix the Kubernetes setup of a payments API before launch: deploys, resources, security, networking, scaling and upgrades.