Why can a cluster add nodes while average CPU usage is under 20%?

The autoscaler reacts to pending pods, and scheduling is based on requests; inflated requests make nodes look full although the capacity is idle.

How should requests be set from usage data?

From observed percentiles over a representative period: CPU near p95 with headroom, memory near p99 plus a buffer, reviewed regularly as usage changes.

What turns right-sized requests into lower bills?

Cluster or node autoscaling that removes now-unneeded nodes, and cost visibility per team so owners keep requests accurate.

Paying for Idle Capacity: Right-Sizing — Kubernetes in Production

Scenario

Example company: the Kubernetes bill doubles in a year while traffic grows 20 %. Nodes look full to the scheduler, so the cluster autoscaler keeps adding them, but average node CPU usage is 18 %.

What you see in production

Most workloads request four to ten times what they use. New pods stay Pending until another node is added, because the reserved capacity is "taken", although it's idle.

The naive code

resources:
  requests: {cpu: "1", memory: 2Gi}     # "to be safe", copied into every service

Reproduce it

Compute right-sized requests from usage percentiles and the nodes saved:

Right-sizing requests from observed usageTry it yourself

import math

# workload: replicas, requested cpu (m), requested memory (Mi), p95 cpu (m), p99 memory (Mi)
workloads = {
    "api":      (6, 1000, 2048, 180, 600),
    "worker":   (4, 1000, 2048, 350, 900),
    "frontend": (3, 500, 1024, 40, 150),
    "cron":     (2, 1000, 2048, 20, 200),
}
NODE_CPU, NODE_MEM = 4000, 16384          # allocatable per node

def nodes_needed(cpu_m, mem_mi):
    return max(math.ceil(cpu_m / NODE_CPU), math.ceil(mem_mi / NODE_MEM))

before_cpu = sum(r * c for r, c, m, pc, pm in workloads.values())
before_mem = sum(r * m for r, c, m, pc, pm in workloads.values())
after = {n: (r, math.ceil(pc * 1.3 / 10) * 10, math.ceil(pm * 1.2 / 64) * 64) for n, (r, c, m, pc, pm) in workloads.items()}
after_cpu = sum(r * c for r, c, m in after.values())
after_mem = sum(r * m for r, c, m in after.values())
for name, (r, c, m) in after.items():
    print(f"{name:9} requests: {workloads[name][1]}m/{workloads[name][2]}Mi -> {c}m/{m}Mi")
print(f"nodes needed for requests: {nodes_needed(before_cpu, before_mem)} before, {nodes_needed(after_cpu, after_mem)} after")

import math

# workload: replicas, requested cpu (m), requested memory (Mi), p95 cpu (m), p99 memory (Mi)
workloads = {
    "api":      (6, 1000, 2048, 180, 600),
    "worker":   (4, 1000, 2048, 350, 900),
    "frontend": (3, 500, 1024, 40, 150),
    "cron":     (2, 1000, 2048, 20, 200),
}
NODE_CPU, NODE_MEM = 4000, 16384          # allocatable per node

def nodes_needed(cpu_m, mem_mi):
    return max(math.ceil(cpu_m / NODE_CPU), math.ceil(mem_mi / NODE_MEM))

before_cpu = sum(r * c for r, c, m, pc, pm in workloads.values())
before_mem = sum(r * m for r, c, m, pc, pm in workloads.values())
after = {n: (r, math.ceil(pc * 1.3 / 10) * 10, math.ceil(pm * 1.2 / 64) * 64) for n, (r, c, m, pc, pm) in workloads.items()}
after_cpu = sum(r * c for r, c, m in after.values())
after_mem = sum(r * m for r, c, m in after.values())
for name, (r, c, m) in after.items():
    print(f"{name:9} requests: {workloads[name][1]}m/{workloads[name][2]}Mi -> {c}m/{m}Mi")
print(f"nodes needed for requests: {nodes_needed(before_cpu, before_mem)} before, {nodes_needed(after_cpu, after_mem)} after")

Output

api       requests: 1000m/2048Mi -> 240m/768Mi
worker    requests: 1000m/2048Mi -> 460m/1088Mi
frontend  requests: 500m/1024Mi -> 60m/192Mi
cron      requests: 1000m/2048Mi -> 30m/256Mi
nodes needed for requests: 4 before, 1 after

Root cause

The scheduler and the cluster autoscaler work on requests, not usage. Inflated requests make nodes look full, so the autoscaler adds nodes that sit mostly idle. Nobody owns revisiting requests after launch, and copying "safe" defaults compounds the waste across every service.

The fix

Measure usage per workload (Prometheus, metrics-server) over weeks, including peaks.
Set requests from percentiles: CPU near p95 with headroom, memory near p99 plus a buffer (memory shortfalls mean OOMKills).
Use VPA in recommendation mode or tools like Goldilocks to propose values; review them in pull requests.
Use cluster autoscaling (Cluster Autoscaler, Karpenter) so freed capacity turns into removed nodes, and consider spot capacity for interruptible workloads.
Report cost per namespace or team (OpenCost) so owners see the effect.

Trade-offs

Tighter requests leave less headroom for spikes; pair with HPA and sensible minimums.
Right-sizing is continuous work; usage changes with features and traffic.
Bin-packing more densely increases the impact of a node failure.

Detect it in production

Requested versus used CPU and memory per namespace (efficiency ratio).
Nodes with high requested but low used resources.
Pending pods while average node utilisation is low.

Key points

Requests reserve capacity, used or not; schedulers and autoscalers see only requests.
Right-size from measured percentiles, memory with more headroom than CPU.
Make savings real with cluster autoscaling and visible with cost reporting.

Further reading: Vertical Pod Autoscaler · OpenCost · Node autoscaling