Restart Loop at Peak Traffic

Scenario

Example team: a Java order service moves from VMs with 8 GB RAM to Kubernetes pods with limits.memory: 1Gi. The JVM flags were copied from the VM setup: -Xmx2g. Load tests passed with small data sets.

What you see in production

At the evening peak pods restart every few minutes. kubectl describe pod shows OOMKilled, exit code 137. Each restart drops in-flight orders and leaves fewer pods for the remaining traffic, so the others die faster. The JVM never logged an OutOfMemoryError: the kernel killed it first.

The naive code

containers:
  - name: orders
    image: registry.example.com/orders:9a1c3e2
    env:
      - name: JAVA_OPTS
        value: "-Xmx2g"           # heap alone may grow past the container limit
    resources:
      limits:
        memory: 1Gi

Reproduce it

Container memory is heap plus everything else: thread stacks, metaspace, direct buffers, the JIT and native libraries. The kernel kills the container when the total crosses the limit, regardless of the runtime's own heap setting:

LIMIT_MB = 1024
NON_HEAP_MB = 300          # metaspace, thread stacks, code cache, direct buffers

def peak_usage(max_heap_mb, live_data_mb):
    heap = min(max_heap_mb, live_data_mb * 2.2)   # the GC lets heap grow well above live data before collecting
    return heap + NON_HEAP_MB

for label, max_heap in [("-Xmx2g", 2048), ("MaxRAMPercentage=65", int(LIMIT_MB * 0.65))]:
    for live in [150, 300, 450]:
        used = peak_usage(max_heap, live)
        state = "OOMKilled (137)" if used > LIMIT_MB else "ok"
        print(f"{label:20} live data {live:3} MB -> container uses {used:6.0f} MB of {LIMIT_MB}: {state}")

-Xmx2g               live data 150 MB -> container uses    630 MB of 1024: ok
-Xmx2g               live data 300 MB -> container uses    960 MB of 1024: ok
-Xmx2g               live data 450 MB -> container uses   1290 MB of 1024: OOMKilled (137)
MaxRAMPercentage=65  live data 150 MB -> container uses    630 MB of 1024: ok
MaxRAMPercentage=65  live data 300 MB -> container uses    960 MB of 1024: ok
MaxRAMPercentage=65  live data 450 MB -> container uses    965 MB of 1024: ok

Root cause

The JVM (and Node.js's V8, .NET, Go) size their heap from flags or from what memory they detect. A fixed -Xmx2g in a 1 GiB container lets the heap grow beyond the limit, and non-heap memory comes on top. The kernel's OOM killer then sends SIGKILL; there's no graceful shutdown and no Java exception. Restarts under load cascade, because the surviving pods receive more traffic.

The fix

1. Size the runtime to the container: for the JVM use -XX:MaxRAMPercentage=60 to 75 (container-aware since Java 10), for Node.js --max-old-space-size below the limit, for .NET the GCHeapHardLimitPercent setting, and for Go GOMEMLIMIT.
2. Set requests and limits from measurements: observe real usage (p95/p99 of working set) under production-like load, set requests.memory near typical usage and limits.memory with headroom. Many teams set requests equal to limits for memory.
3. Load test with realistic data volumes, not just request rates.
4. Add capacity first during an incident (more replicas or a higher limit), then fix the sizing.

env:
  - name: JAVA_TOOL_OPTIONS
    value: "-XX:MaxRAMPercentage=65 -XX:+ExitOnOutOfMemoryError"
resources:
  requests: { cpu: 500m, memory: 1Gi }
  limits:   { memory: 1Gi }

Trade-offs

• A smaller heap means more frequent garbage collection; tune with GC logs rather than guessing.
• Generous limits waste node capacity; tight ones risk kills. Vertical Pod Autoscaler recommendations help find the middle.
• -XX:+ExitOnOutOfMemoryError turns a sick JVM into a clean restart, which is good only if you have enough replicas.

Detect it in production

• Alert on container restarts and OOMKilled reasons (kube_pod_container_status_last_terminated_reason), not just on pod availability.
• Track memory working set as a percentage of the limit per container.
• GC metrics (heap after collection) show whether live data is growing (a leak) or the heap is just sized wrong.

Key points

• The kernel enforces container memory limits with SIGKILL (exit 137), not a runtime exception.
• Configure runtimes relative to the container limit; remember non-heap memory.
• Set requests and limits from measured usage, and alert on restarts.