Kubernetes cookbook · Intermediate

Roll out without dropping requests

Deploys and scale-downs cause a burst of 502 errors.

Create a free account to save this page, take notes and track your progress. Sign in

Approach

Readiness gates traffic to new pods, maxUnavailable: 0 keeps capacity, and a preStop delay plus graceful SIGTERM handling lets old pods finish while load balancers stop sending them traffic.

Solution

apiVersion: apps/v1
kind: Deployment
metadata: {name: web}
spec:
  replicas: 4
  minReadySeconds: 10
  strategy:
    type: RollingUpdate
    rollingUpdate: {maxUnavailable: 0, maxSurge: 25%}
  selector: {matchLabels: {app: web}}
  template:
    metadata: {labels: {app: web}}
    spec:
      terminationGracePeriodSeconds: 40
      containers:
        - name: web
          image: ghcr.io/acme/web:3.2.0
          readinessProbe: {httpGet: {path: /ready, port: 8080}, periodSeconds: 3}
          lifecycle: {preStop: {sleep: {seconds: 8}}}

Pitfalls

  • Exiting immediately on SIGTERM while traffic is still being routed to the pod.
  • Grace period shorter than preStop plus drain time.

Variations

  • Progressive delivery (canary) with Argo Rollouts or Gateway API weights for risky changes.
  • Connection draining settings on cloud load balancers (deregistration delay).

Further reading: Pod termination

Reference: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination

More Kubernetes recipes

Was this page helpful?