Every Deploy Drops Requests

Scenario

Example team: a Node.js API deploys 10 times a day with a rolling update. The Dockerfile ends with CMD npm start. There's a liveness probe but no readiness probe.

What you see in production

Every deploy produces a burst of 502s and connection resets, a few hundred requests each time. Users retry, so nobody escalates, but error budgets drain on deploy days, and payment webhooks occasionally fail. Pods take exactly 30 seconds to terminate.

The naive code

CMD npm start              # shell form: /bin/sh is PID 1, npm is a child, node a grandchild

// server.js: no SIGTERM handler
app.listen(8080);

Reproduce it

What a terminating process should do when it gets SIGTERM, shown in bash:

inflight=3
on_term() {
  echo "SIGTERM: marked not ready, stop accepting new connections"
  while (( inflight > 0 )); do
    echo "  finishing request ($inflight in flight)"
    inflight=$((inflight - 1))
  done
  echo "all requests done, exiting 0"
  exit 0
}
trap on_term TERM
echo "serving..."
kill -TERM $$

serving...
SIGTERM: marked not ready, stop accepting new connections
  finishing request (3 in flight)
  finishing request (2 in flight)
  finishing request (1 in flight)
all requests done, exiting 0

And why timing matters: when Kubernetes deletes a pod it sends SIGTERM and removes the pod from endpoints at the same time, but load balancers and kube-proxy need a few seconds to stop sending traffic:

RPS = 200                # requests per second to one pod
STARTUP_S = 3            # seconds a new pod needs before it can serve
LB_UPDATE_S = 4          # seconds until endpoints/load balancers stop routing to a terminating pod
INFLIGHT = 30            # requests in progress at any moment
LONG_RUNNING = 5         # uploads and reports still running 30 s after SIGTERM

setups = {
    "naive: no readiness probe, SIGTERM ignored": {
        "new pod routed before ready": RPS * STARTUP_S,
        "old pod SIGKILLed at 30 s (long requests cut)": LONG_RUNNING},
    "readiness probe, app exits at once on SIGTERM": {
        "requests still routed to the stopped pod": RPS * LB_UPDATE_S,
        "in-flight requests aborted": INFLIGHT},
    "readiness + preStop sleep 5 + drain on SIGTERM": {},
}
for name, failures in setups.items():
    print(f"{name}: {sum(failures.values())} failed requests per pod replaced")
    for cause, n in failures.items():
        print(f"    {n:4}  {cause}")

naive: no readiness probe, SIGTERM ignored: 605 failed requests per pod replaced
     600  new pod routed before ready
       5  old pod SIGKILLed at 30 s (long requests cut)
readiness probe, app exits at once on SIGTERM: 830 failed requests per pod replaced
     800  requests still routed to the stopped pod
      30  in-flight requests aborted
readiness + preStop sleep 5 + drain on SIGTERM: 0 failed requests per pod replaced

Root cause

Two problems combine. Without a readiness probe, a new pod is added to the Service as soon as its container starts, so it receives requests before the app is listening: those are the 502s and resets. On the way out, a shell-form CMD makes /bin/sh PID 1, which doesn't forward SIGTERM, so the app never learns it's being stopped and is SIGKILLed after the 30-second grace period, cutting off long-running requests (and making every pod take 30 s to stop). Fixing only the signal handling isn't enough either: endpoints and load balancers update asynchronously, so an app that exits immediately on SIGTERM still receives new requests for a few seconds.

The fix

1. Exec-form CMD so the app is PID 1 (or use a tiny init like tini): CMD ["node", "server.js"].
2. Handle SIGTERM: stop accepting new connections, finish in-flight requests, close pools, exit.
3. Delay shutdown briefly with a preStop hook so endpoints update first.
4. Readiness probe so new pods get traffic only when ready.

const server = app.listen(8080);
process.on("SIGTERM", () => {
  server.close(() => process.exit(0));          // stop accepting, wait for in-flight requests
  setTimeout(() => process.exit(1), 25_000).unref();
});

spec:
  terminationGracePeriodSeconds: 30
  containers:
    - name: api
      lifecycle:
        preStop:
          exec: { command: ["sleep", "5"] }      # let endpoints and load balancers catch up
      readinessProbe:
        httpGet: { path: /readyz, port: 8080 }
        periodSeconds: 5

CMD ["node", "server.js"]

Trade-offs

• Graceful shutdown makes deploys slower (seconds per pod); worth it compared with dropped requests.
• preStop needs a sleep binary in the image (distroless images may need a native sleep in the app instead).
• Long-lived connections (WebSockets, gRPC streams) need explicit draining and client reconnect logic.

Detect it in production

• Error-rate panels annotated with deploy markers: spikes at every deploy mean shutdown isn't graceful.
• Pod termination durations: exactly the grace period means SIGTERM was ignored.
• Load balancer 502/504 counts per target during rollouts.

Key points

• Shell-form CMD hides SIGTERM from the app; use exec form or an init.
• Handle SIGTERM by draining, and wait a few seconds (preStop) for endpoints to update.
• Readiness probes keep new pods out of rotation until they're ready.