Out of Memory: Limits, Swapping and the OOM Killer

Scenario

A batch job loads a whole export into memory. On a busy night it gets killed with no error in its own logs; the exit code is 137. Another night the whole server becomes unresponsive for minutes before recovering.

What you see in production

• Processes exit with code 137 (128 + 9, SIGKILL) and no application error.
• dmesg or the journal shows Out of memory: Killed process ....
• Kubernetes reports OOMKilled; Docker shows "OOMKilled": true.
• Heavy swapping makes everything slow before the kill.

The naive code

rows = open("export.csv").read().splitlines()   # the whole file in memory
totals = [process(r) for r in rows]             # and a second full-size list

Reproduce it

A memory limit on one process (here prlimit --as, the address-space limit) makes allocation fail inside the process, where the app can see it:

prlimit --as=300000000 python3 - <<'EOF'
try:
    blocks = []
    for i in range(100):
        blocks.append(bytearray(10_000_000))      # 10 MB each
except MemoryError:
    print(f"MemoryError after allocating about {len(blocks) * 10} MB")
EOF
bash -c 'kill -KILL $$' ; echo "a process killed by SIGKILL exits with status $?"

MemoryError after allocating about 280 MB
a process killed by SIGKILL exits with status 137

Root cause

When the system (or a cgroup, as in containers) has no memory left to give, the kernel's OOM killer picks a process, usually the largest, and sends SIGKILL. The process gets no chance to log or clean up, which is why the only evidence is exit code 137 and kernel logs. Processes that read whole inputs into memory, or grow caches without bounds, are the usual cause.

The fix

Process data in streams with bounded memory, and set explicit limits so failures are predictable and contained:

python3 -c 'import sys; [print(f"{i},{i % 7}") for i in range(500_000)]' > export.csv
/usr/bin/time -f "whole file: %M KB peak" python3 -c '
rows = open("export.csv").read().splitlines()
print(sum(int(r.split(",")[1]) for r in rows))' 2>&1
/usr/bin/time -f "streaming:  %M KB peak" python3 -c '
with open("export.csv") as f:
    print(sum(int(line.split(",")[1]) for line in f))' 2>&1

1499994
whole file: 48348 KB peak
1499994
streaming:  10392 KB peak

• Stream inputs (iterate lines, use generators, paginate queries) instead of loading everything.
• Bound caches and queues.
• Set memory limits per service: MemoryMax= in systemd, --memory in Docker, resources.limits.memory in Kubernetes, so one process can't take down the host.
• For JVM, Node and similar runtimes, set heap sizes relative to the container limit.

Trade-offs

• Limits that are too tight cause OOM kills under normal peaks; base them on measured usage plus headroom.
• Swap can absorb short spikes but turns memory pressure into slowness; many servers disable it or keep it small.
• Restarting on OOM hides leaks; track memory growth over time.

Detect it in production

• journalctl -k | grep -i 'killed process' or dmesg -T | grep -i oom.
• Exit code 137 and OOMKilled status in container platforms.
• Memory usage graphs per process/container, with alerts on approaching limits and on steady growth.

Key points

• The OOM killer sends SIGKILL: no logs, exit code 137.
• Stream data and bound caches to keep memory flat.
• Set per-service memory limits and watch usage against them.

Further reading: proc(5): oom_score · systemd.resource-control: MemoryMax · Kubernetes: resource management