Pods Stopped Scheduling

Scenario

Example team: an EKS cluster runs in three private /24 subnets, one per availability zone, created when the cluster had 6 nodes. With the AWS VPC CNI, every pod gets a real VPC IP address. Black Friday scaling takes the cluster to 40 nodes.

What you see in production

New pods stay Pending or fail with failed to assign an IP address to container. The autoscaler adds nodes, which makes it worse: each new node reserves warm IPs too. Lambda functions attached to the same subnets start failing with ENI errors. CPU and memory are fine; the cluster has run out of addresses.

The naive code

module "vpc" {
  source          = "terraform-aws-modules/vpc/aws"
  cidr            = "10.20.0.0/16"
  private_subnets = ["10.20.1.0/24", "10.20.2.0/24", "10.20.3.0/24"]   # 251 usable each
}

Reproduce it

Count the addresses a growing cluster needs per zone (nodes + pods + the CNI's warm pool), against what a subnet really offers:

import ipaddress

AWS_RESERVED = 5                      # network, router, DNS, future use, broadcast

def usable(cidr):
    return ipaddress.ip_network(cidr).num_addresses - AWS_RESERVED

def needed(nodes, pods_per_node, warm_per_node):
    return nodes + nodes * pods_per_node + nodes * warm_per_node

zones = 3
for nodes in [6, 20, 40]:
    per_zone = -(-nodes // zones)     # ceiling division
    need = needed(per_zone, pods_per_node=25, warm_per_node=8)
    for cidr in ["10.20.1.0/24", "10.20.16.0/20"]:
        status = "ok" if need <= usable(cidr) else "EXHAUSTED"
        print(f"{nodes:2} nodes: {per_zone:2}/zone need {need:4} IPs, {cidr:14} has {usable(cidr):4} -> {status}")

 6 nodes:  2/zone need   68 IPs, 10.20.1.0/24   has  251 -> ok
 6 nodes:  2/zone need   68 IPs, 10.20.16.0/20  has 4091 -> ok
20 nodes:  7/zone need  238 IPs, 10.20.1.0/24   has  251 -> ok
20 nodes:  7/zone need  238 IPs, 10.20.16.0/20  has 4091 -> ok
40 nodes: 14/zone need  476 IPs, 10.20.1.0/24   has  251 -> EXHAUSTED
40 nodes: 14/zone need  476 IPs, 10.20.16.0/20  has 4091 -> ok

Root cause

With VPC-native pod networking (AWS VPC CNI, Azure CNI with VNet IPs, GKE alias IPs), pods consume addresses from the VPC like VMs do. Subnet sizes chosen for "a few dozen servers" can't hold hundreds of pods plus warm pools, and a subnet can't be resized after creation. Other address consumers share the same subnets: load balancer nodes, Lambda/Functions VNet integration, endpoints.

The fix

1. Size subnets for the workload. For Kubernetes use /19 or /20 per zone at least; keep small subnets for things that really are small.
2. Give pods their own range. On AWS, add a secondary VPC CIDR from 100.64.0.0/10 (carrier-grade NAT space, not routable on the internet) and use custom networking so pods take IPs from it while nodes stay in the primary range.
3. Use prefix delegation to assign /28 prefixes to node ENIs, which raises pods per node and reduces API calls (it needs contiguous free /28 blocks, so fragmented subnets still run out).
4. Or use an overlay where pods don't use VPC addresses at all (Azure CNI Overlay, Cilium overlay, GKE with secondary ranges sized for pods).

resource "aws_vpc_ipv4_cidr_block_association" "pods" {
  vpc_id     = aws_vpc.main.id
  cidr_block = "100.64.0.0/16"            # pod-only range, never routed outside the VPC
}

On Azure, choose Azure CNI Overlay for large clusters; on GKE, plan the pod and service secondary ranges before creating the cluster, since they are hard to change later.

Trade-offs

• Bigger subnets use more of the VPC range up front, so plan the VPC as a /16 or larger.
• Pod-only 100.64.0.0/10 ranges aren't reachable from outside the VPC without NAT; that is usually what you want.
• Overlays save addresses but add encapsulation and make pods invisible to VPC-level tools such as security groups and flow logs.

Detect it in production

• Alarm on available IPs per subnet (AWS publishes AvailableIpAddressCount on every subnet; CloudWatch/VPC IPAM can track utilisation).
• Kubernetes events FailedCreatePodSandBox with IP assignment errors; aws-node (VPC CNI) metrics for IP pool usage.
• Capacity reviews include IP headroom, not just CPU and memory.

Key points

• VPC-native pods consume subnet addresses; small subnets run out long before compute does.
• AWS reserves 5 addresses per subnet and subnets can't be resized, so size for growth.
• Use secondary 100.64.0.0/10 ranges with custom networking, prefix delegation or an overlay.