Node Drains That Take a Service Down

Scenario

Example company: the platform team upgrades node pools on Tuesday afternoons. A 3-replica checkout API has all its pods on the same two nodes, both of which get drained in the first batch.

What you see in production

Checkout is down for four minutes during the upgrade. All three pods were evicted within seconds of each other and scheduled onto new nodes, where they spent minutes pulling images and warming up.

The naive code

kind: Deployment
spec:
  replicas: 3          # no PodDisruptionBudget, no topology spread: replicas may share nodes

Reproduce it

How many replicas survive a drain of two nodes, depending on placement and a PodDisruptionBudget:

def drain(placement, nodes_to_drain, min_available=None):
    pods = dict(placement)                       # pod -> node
    available = len(pods)
    evicted, blocked = [], []
    for node in nodes_to_drain:
        for pod, n in sorted(pods.items()):
            if n != node:
                continue
            if min_available is not None and available - 1 < min_available:
                blocked.append(pod)              # eviction refused until a replacement is ready
                continue
            evicted.append(pod)
            available -= 1
    return available, evicted, blocked

packed = {"checkout-a": "node-1", "checkout-b": "node-1", "checkout-c": "node-2"}
spread = {"checkout-a": "node-1", "checkout-b": "node-2", "checkout-c": "node-3"}
for name, placement, pdb in [("packed, no PDB", packed, None), ("packed, PDB minAvailable=2", packed, 2),
                             ("spread over zones, PDB minAvailable=2", spread, 2)]:
    avail, ev, bl = drain(placement, ["node-1", "node-2"], pdb)
    print(f"{name:38} available during drain: {avail}  evicted: {len(ev)}  waiting: {len(bl)}")

packed, no PDB                         available during drain: 0  evicted: 3  waiting: 0
packed, PDB minAvailable=2             available during drain: 2  evicted: 1  waiting: 2
spread over zones, PDB minAvailable=2  available during drain: 2  evicted: 1  waiting: 1

Root cause

A drain evicts every pod on a node through the Eviction API. Without a PodDisruptionBudget, nothing limits how many of a service's pods are voluntarily disrupted at the same time. Without topology spread constraints, the scheduler may place replicas on the same node or zone, so a single drain (or a single node failure) takes several out together.

The fix

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: {name: checkout}
spec:
  minAvailable: 2                  # or maxUnavailable: 1
  selector: {matchLabels: {app: checkout}}
---
# in the Deployment's pod template
topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway
    labelSelector: {matchLabels: {app: checkout}}

• PDBs make drains wait until replacements are ready elsewhere.
• Spread replicas across nodes and zones; combine with enough replicas that losing one zone is survivable.
• Pre-pull images or keep them small so rescheduled pods start fast.

Trade-offs

• A PDB that allows zero disruptions (maxUnavailable: 0, or minAvailable equal to replicas) blocks drains forever and stalls upgrades.
• PDBs only cover voluntary disruptions; node crashes ignore them, which is why spreading matters.
• Spreading across zones can add cross-zone traffic costs and latency.

Detect it in production

• Workloads with more than one replica and no PDB (audit or policy).
• Replicas of one Deployment sharing a node or zone.
• Drains that stall because a PDB can never be satisfied.

Key points

• Drains evict pods; PDBs limit how many go at once.
• Topology spread constraints keep replicas on different nodes and zones.
• Never configure a PDB that allows zero disruptions.

Further reading: Specifying a Disruption Budget · Pod topology spread constraints