Why might an HPA show 50% CPU while the app is saturated?

Utilisation is measured against the CPU request; if the request is much higher than what the app can actually use, the percentage stays low even at saturation.

What does the HPA scale-down stabilisation window do?

It uses the highest recommendation over the window before scaling down, preventing replicas from dropping on every brief lull.

When should you scale on something other than CPU?

When CPU doesn't track load, for example I/O-bound services or queue consumers; scale on requests per second, latency or queue length with custom or external metrics such as KEDA.

Autoscaling That Flaps or Never Scales — Kubernetes in Production

Scenario

Example company: an API has an HPA targeting 70 % CPU. During a traffic peak it doesn't scale and latency triples. On normal days it oscillates between 3 and 9 replicas every few minutes.

What you see in production

During the peak, kubectl get hpa shows CPU 50 %/70 % while pods are visibly saturated. On quiet days, replica counts saw-tooth and new pods' cold starts cause latency spikes.

The naive code

resources:
  requests: {cpu: "2"}            # copied from a template; the app uses about 300m at normal load
---
spec:
  metrics: [{type: Resource, resource: {name: cpu, target: {type: Utilization, averageUtilization: 70}}}]
  # no behavior: default scale-down after a 5-minute window, no limits on step size

Reproduce it

The HPA formula with a request far above real usage, and with a sensible request plus a stabilisation window:

desired = ceil(replicas x utilisation / target)Try it yourself

import math

SATURATION_M = 1000          # one pod can't use more than about 1 CPU (its threads are busy)

def step(replicas, demand_m, request_m, target=70):
    per_pod = min(demand_m / replicas, SATURATION_M)          # CPU each pod actually uses
    utilisation = 100 * per_pod / request_m                    # what the HPA sees
    desired = max(2, math.ceil(replicas * utilisation / target))
    saturated = demand_m / replicas > SATURATION_M             # more work than the pods can do
    return desired, utilisation, saturated

demand = [900, 2700, 4800, 4800, 2700, 900]                    # total CPU the traffic needs (millicores)
for request in (2000, 400):
    replicas, out = 3, []
    for d in demand:
        new, util, saturated = step(replicas, d, request)
        out.append(f"{util:.0f}%{'!' if saturated else ''}->{new}")
        replicas = new
    print(f"request {request:4}m: " + ", ".join(out))
print("('!' = pods saturated: users see high latency)")

# scale-down stabilisation: act on the highest recommendation of the last 3 evaluations
recs = [9, 3, 9, 3, 3, 3, 3]
smoothed = [max(recs[max(0, i - 2): i + 1]) for i in range(len(recs))]
print("raw recommendations:", recs, "-> with a scale-down window:", smoothed)

import math

SATURATION_M = 1000          # one pod can't use more than about 1 CPU (its threads are busy)

def step(replicas, demand_m, request_m, target=70):
    per_pod = min(demand_m / replicas, SATURATION_M)          # CPU each pod actually uses
    utilisation = 100 * per_pod / request_m                    # what the HPA sees
    desired = max(2, math.ceil(replicas * utilisation / target))
    saturated = demand_m / replicas > SATURATION_M             # more work than the pods can do
    return desired, utilisation, saturated

demand = [900, 2700, 4800, 4800, 2700, 900]                    # total CPU the traffic needs (millicores)
for request in (2000, 400):
    replicas, out = 3, []
    for d in demand:
        new, util, saturated = step(replicas, d, request)
        out.append(f"{util:.0f}%{'!' if saturated else ''}->{new}")
        replicas = new
    print(f"request {request:4}m: " + ", ".join(out))
print("('!' = pods saturated: users see high latency)")

# scale-down stabilisation: act on the highest recommendation of the last 3 evaluations
recs = [9, 3, 9, 3, 3, 3, 3]
smoothed = [max(recs[max(0, i - 2): i + 1]) for i in range(len(recs))]
print("raw recommendations:", recs, "-> with a scale-down window:", smoothed)

Output

request 2000m: 15%->2, 50%!->2, 50%!->2, 50%!->2, 50%!->2, 22%->2
request  400m: 75%->4, 169%->10, 120%->18, 67%->18, 38%->10, 22%->4
('!' = pods saturated: users see high latency)
raw recommendations: [9, 3, 9, 3, 3, 3, 3] -> with a scale-down window: [9, 9, 9, 9, 9, 3, 3]

Root cause

CPU utilisation is measured relative to the request. A request of 2 CPUs for an app that saturates at 1 CPU means the HPA sees 50 % at the moment the app is overloaded. Conversely, spiky load with no stabilisation makes recommendations jump up and down, so replicas follow every spike and every lull.

The fix

Set CPU requests from measured usage so utilisation reflects real saturation.
Choose a target with headroom for scale-up delay (pods need time to start and warm up).
Tune behavior: a scale-down stabilisation window (for example 300 s), limits on how many pods to remove per minute, fast scale-up.
Scale on the metric that actually tracks load when CPU doesn't (requests per second, queue length) using custom or external metrics (KEDA).
Keep minReplicas high enough to survive a spike while new pods start.

behavior:
  scaleUp:   {stabilizationWindowSeconds: 0, policies: [{type: Percent, value: 100, periodSeconds: 30}]}
  scaleDown: {stabilizationWindowSeconds: 300, policies: [{type: Pods, value: 2, periodSeconds: 60}]}

Trade-offs

Lower targets and higher minimums cost more but absorb spikes.
Long scale-down windows keep idle pods longer.
Custom metrics add moving parts (adapters, Prometheus availability).

Detect it in production

HPA at max replicas (or never moving) during incidents.
Replica count oscillation (changes per hour).
Utilisation versus latency: high latency at low "utilisation" means requests are wrong.

Key points

HPA utilisation is usage divided by request; wrong requests mislead it.
Use behavior settings to scale up fast and down slowly.
Scale on the metric that tracks load; CPU isn't always it.

Further reading: Horizontal Pod Autoscaling · KEDA