Why do pods without resource requests cause problems even when nodes look fine?

The scheduler treats them as needing nothing, so it packs too many pods onto nodes; under real load the node runs out of memory and pods are evicted or killed.

What's the difference between being OOMKilled and being evicted?

OOMKilled: the kernel kills a container that exceeds its own memory limit. Eviction: the kubelet removes pods when the node itself is under resource pressure, starting with BestEffort and over-request pods.

How do you enforce sensible resource defaults across teams?

LimitRange objects set default requests and limits per namespace, ResourceQuota caps totals, and admission policies reject pods without resources.

OOMKills, Evictions and Noisy Neighbours — Kubernetes in Production

Scenario

Example company: teams deploy without resource settings because "Kubernetes will figure it out". One night a batch job's memory grows, and several unrelated services restart.

What you see in production

Pods restart with OOMKilled, some are evicted with The node was low on resource: memory, and latency spikes for services that share nodes with the batch job. The scheduler keeps packing more pods onto already busy nodes because their declared requests are zero.

The naive code

containers:
  - name: report-job
    image: registry.example.com/report:5.1.0
    # no resources: requests 0 (scheduler thinks it's free), no limit (can use the whole node)

Reproduce it

A pod with a memory limit below what it uses, and what the scheduler believes about pods without requests:

An OOMKill, and how requests drive scheduling

kubectl create namespace prod-res > /dev/null
trap 'kubectl delete namespace prod-res --wait=false > /dev/null' EXIT
kubectl -n prod-res run greedy --restart=Never --image=busybox:1.37 --overrides='{"spec":{"containers":[{"name":"greedy","image":"busybox:1.37","command":["sh","-c","tail /dev/zero"],"resources":{"requests":{"memory":"32Mi"},"limits":{"memory":"32Mi"}}}]}}' > /dev/null
kubectl -n prod-res run lazy --image=busybox:1.37 -- sleep 3600 > /dev/null
kubectl -n prod-res wait --for=jsonpath='{.status.phase}'=Failed pod/greedy --timeout=120s > /dev/null
echo "greedy: $(kubectl -n prod-res get pod greedy -o jsonpath='{.status.containerStatuses[0].state.terminated.reason}')"
echo "lazy (no requests) QoS class: $(kubectl -n prod-res get pod lazy -o jsonpath='{.status.qosClass}') -> first to be evicted under node pressure"
alloc=$(kubectl get nodes -o jsonpath='{.items[0].status.allocatable.memory}')
echo "node allocatable memory: $alloc; requested by 'lazy': $(kubectl -n prod-res get pod lazy -o jsonpath='{.spec.containers[0].resources.requests.memory}')<none>"

kubectl create namespace prod-res > /dev/null
trap 'kubectl delete namespace prod-res --wait=false > /dev/null' EXIT
kubectl -n prod-res run greedy --restart=Never --image=busybox:1.37 --overrides='{"spec":{"containers":[{"name":"greedy","image":"busybox:1.37","command":["sh","-c","tail /dev/zero"],"resources":{"requests":{"memory":"32Mi"},"limits":{"memory":"32Mi"}}}]}}' > /dev/null
kubectl -n prod-res run lazy --image=busybox:1.37 -- sleep 3600 > /dev/null
kubectl -n prod-res wait --for=jsonpath='{.status.phase}'=Failed pod/greedy --timeout=120s > /dev/null
echo "greedy: $(kubectl -n prod-res get pod greedy -o jsonpath='{.status.containerStatuses[0].state.terminated.reason}')"
echo "lazy (no requests) QoS class: $(kubectl -n prod-res get pod lazy -o jsonpath='{.status.qosClass}') -> first to be evicted under node pressure"
alloc=$(kubectl get nodes -o jsonpath='{.items[0].status.allocatable.memory}')
echo "node allocatable memory: $alloc; requested by 'lazy': $(kubectl -n prod-res get pod lazy -o jsonpath='{.spec.containers[0].resources.requests.memory}')<none>"

Output

greedy: 
lazy (no requests) QoS class:  -> first to be evicted under node pressure
node allocatable memory: 18516204Ki; requested by 'lazy': <none>

Run this on your computer — see “Get Started” for setup.

Root cause

The scheduler only knows what pods request. Pods without requests look free, so nodes are overcommitted. Pods without limits can grow until the node runs short, and then the kubelet evicts pods, starting with BestEffort (no requests) and pods using more than they requested. A container above its own memory limit is killed by the kernel (OOMKilled, exit code 137).

The fix

resources:
  requests: {cpu: 200m, memory: 512Mi}    # from observed usage, e.g. p95 plus headroom
  limits:   {memory: 768Mi}              # cap memory; usually no CPU limit (see the throttling lesson)

Measure real usage (metrics-server, Prometheus, VPA in recommendation mode) and set requests near steady-state usage.
Set memory limits for every container; for critical services consider requests = limits (Guaranteed QoS).
Enforce defaults per namespace with a LimitRange and budgets with a ResourceQuota.
Make runtimes respect container limits (JVM -XX:MaxRAMPercentage, Node --max-old-space-size).

Trade-offs

Generous requests waste money (reserved but idle capacity); tight ones cause evictions. Revisit them regularly.
Memory limits turn leaks into restarts (better than taking down the node, but restarts hurt too).
Guaranteed QoS is the most protected but least efficient.

Detect it in production

Alerts on OOMKilled restarts and eviction events.
Pods without requests or limits (admission policy or audit report).
Usage-to-request ratios per workload (over- and under-provisioning).

Key points

Requests drive scheduling; missing requests overcommit nodes.
Memory limits cap damage; exceeding them means OOMKilled.
Set both from measurements and enforce defaults with LimitRanges and quotas.

Further reading: Resource management for pods · Node-pressure eviction