Dropped Requests When Pods Shut Down

Scenario

Example app: a Node.js API with readiness probes and a correct rolling update strategy. Deploys still produce a small burst of connection reset errors, and so does every scale-down by the autoscaler.

What you see in production

About 0.5 % of requests fail during deploys, always within a second or two of a pod starting to terminate. The application logs show the process exiting immediately on SIGTERM.

The naive code

process.on("SIGTERM", () => process.exit(0));   // "graceful": exit right away

Reproduce it

Termination starts two things in parallel: the kubelet sends SIGTERM, and the endpoint removal propagates to every node's kube-proxy and to load balancers. A simulation of the race:

import random
random.seed(4)

def simulate(prestop_delay_s, requests=10_000):
    # time (s) for the endpoint removal to reach each client path after termination starts
    propagation = [random.uniform(0.2, 3.0) for _ in range(requests)]
    arrival = [random.uniform(0, 3.0) for _ in range(requests)]       # request arrival time after termination starts
    stops_accepting = prestop_delay_s                               # app closes its listener after preStop
    failed = 0
    for p, a in zip(propagation, arrival):
        routed_to_old_pod = a < p                                   # endpoint still listed at arrival
        if routed_to_old_pod and a >= stops_accepting:
            failed += 1
    return failed / requests

print(f"exit immediately:   {simulate(0):.1%} of requests in the first 3 s fail")
print(f"preStop sleep 5 s: {simulate(5):.1%} fail")

exit immediately:   53.1% of requests in the first 3 s fail
preStop sleep 5 s: 0.0% fail

Root cause

Removing a terminating pod from Service endpoints is asynchronous: kube-proxy rules, ingress controllers and cloud load balancers update over the next seconds. Meanwhile the container has already received SIGTERM. An app that exits immediately refuses connections that are still being routed to it.

The fix

spec:
  terminationGracePeriodSeconds: 45
  containers:
    - name: api
      lifecycle:
        preStop:
          sleep: {seconds: 5}      # keep serving while load balancers catch up

process.on("SIGTERM", () => {
  server.close(() => process.exit(0));                // stop accepting, finish in-flight requests
  setTimeout(() => process.exit(1), 30_000).unref();  // safety net below the grace period
});

• The preStop hook runs before SIGTERM is sent, so the pod keeps serving during propagation.
• The grace period must cover preStop + draining.
• Use keep-alive-aware shutdown so idle connections are closed.

Trade-offs

• Longer termination slows rollouts and scale-downs.
• A preStop delay is a heuristic; measure your propagation time (load balancer deregistration delays can be 10 s or more).
• Long-lived connections (WebSockets, gRPC streams) need explicit drain and reconnect logic on clients.

Detect it in production

• Errors on the load balancer correlated with pod termination events.
• Application logs: requests received after SIGTERM.
• Grace period exceeded (exit code 137 on terminating pods).

Key points

• Endpoint removal and SIGTERM happen at the same time; propagation takes seconds.
• Add a preStop delay, then stop accepting and drain in-flight requests.
• Size terminationGracePeriodSeconds to cover both.

Further reading: Pod termination · Container lifecycle hooks