Low CPU, High Latency

Scenario

Example team: an API pod has limits.cpu: 500m. Dashboards show average CPU usage around 30% of the limit, yet p99 latency is 400 ms while p50 is 20 ms. Adding replicas barely helps.

What you see in production

Latency spikes line up with bursts: a request that needs 80 ms of CPU sometimes takes 250 ms. The metric container_cpu_cfs_throttled_periods_total is climbing on every pod, but nobody alerts on it.

The naive code

resources:
  requests: { cpu: 250m }
  limits:   { cpu: 500m }    # 50 ms of CPU time per 100 ms period, across all threads

Reproduce it

The Linux CFS quota gives a container a CPU-time budget per 100 ms period. A burst of parallel work uses the whole budget early in the period, and the container is paused until the next period starts:

PERIOD_MS = 100

def run(cpu_needed_ms, threads, limit_cores):
    quota = limit_cores * PERIOD_MS            # CPU-ms allowed per period across all threads
    elapsed, remaining, throttled = 0, cpu_needed_ms, 0
    while remaining > 0:
        burst = min(remaining, quota, threads * PERIOD_MS)
        remaining -= burst
        busy_ms = burst / threads              # wall time to use the burst with N threads
        if remaining > 0:
            elapsed += PERIOD_MS               # wait for the next period
            throttled += 1
        else:
            elapsed += busy_ms
    return elapsed, throttled

for limit in [0.5, 1.0, 2.0, None]:
    if limit is None:
        print(f"no CPU limit    -> request takes {80 / 4:5.0f} ms, throttled 0 periods")
        continue
    wall, throttled = run(cpu_needed_ms=80, threads=4, limit_cores=limit)
    print(f"limit {limit:3} cores -> request takes {wall:5.0f} ms, throttled {throttled} period(s)")

limit 0.5 cores -> request takes   108 ms, throttled 1 period(s)
limit 1.0 cores -> request takes    20 ms, throttled 0 period(s)
limit 2.0 cores -> request takes    20 ms, throttled 0 period(s)
no CPU limit    -> request takes    20 ms, throttled 0 periods

Root cause

CPU limits are enforced by the CFS bandwidth controller: once a container uses its quota in a 100 ms period, all its threads wait for the next period. Multi-threaded runtimes (JVM GC threads, Go's goroutines, Node's libuv pool) burn through quota quickly in bursts. Average utilisation hides it, because it's measured over seconds. Runtimes that size thread pools by the host's core count (older JVMs, Go before container-aware GOMAXPROCS) make it worse.

The fix

• Set CPU requests from measurement and consider removing CPU limits for latency-sensitive services: requests guarantee a share when the node is busy, and without a limit the container can use idle CPU for bursts. Keep memory limits.
• If limits are required (multi-tenant clusters, cost control), set them well above typical usage and alert on throttling.
• Make runtimes container-aware: modern JVMs read the CPU quota; set GOMAXPROCS to match (newer Go versions do it automatically; older ones need automaxprocs).

resources:
  requests: { cpu: "1", memory: 1Gi }
  limits:   { memory: 1Gi }     # no CPU limit: bursts use spare node CPU

Trade-offs

• Without CPU limits, a misbehaving pod can use spare CPU that other pods expected for their own bursts; requests still protect each pod's guaranteed share.
• Higher requests reserve more capacity and cost more; that's the honest price of the latency you want.
• Some platforms enforce limits by policy; work with the platform team on exceptions for latency-sensitive services.

Detect it in production

• Throttling ratio: rate(container_cpu_cfs_throttled_periods_total) / rate(container_cpu_cfs_periods_total); alert above ~5-10% for latency-critical services.
• Compare p99 latency with throttling; correlated spikes confirm the cause.
• Per-container CPU usage at 1-second resolution during load tests.

Key points

• CPU limits are a per-100 ms budget; bursts get paused even at low average usage.
• Prefer accurate CPU requests and no CPU limit for latency-sensitive services; always keep memory limits.
• Monitor throttled periods, not just CPU percentage.