Too Many Open Files

Scenario

Under load, an API starts failing with Too many open files. Restarting fixes it for a few hours. Someone raises the limit to a million, and the errors return a week later.

What you see in production

• Errors such as EMFILE: too many open files or accept: too many open files.
• The process's open descriptor count grows steadily over time.
• New connections fail while existing ones keep working.

The naive code

def read_config(path):
    f = open(path)          # never closed: one descriptor leaks per call
    return f.read()

Reproduce it

A low limit makes the leak visible quickly:

touch config.ini
ulimit -n 50
python3 - <<'EOF'
leaked = []
try:
    for i in range(100):
        leaked.append(open("config.ini"))      # opened, never closed
except OSError as e:
    print(f"failed after {len(leaked)} files: {e.strerror}")
print(f"open descriptors: {len(leaked) + 3} (stdin, stdout, stderr and the leaked files)")
EOF

failed after 47 files: Too many open files
open descriptors: 50 (stdin, stdout, stderr and the leaked files)

Root cause

Every open file, socket and pipe uses a file descriptor, and each process has a limit (ulimit -n, RLIMIT_NOFILE). Code paths that open without closing (missing with, unclosed HTTP responses, connection pools without limits) leak descriptors until the limit is reached.

The fix

Close resources deterministically, watch the descriptor count, and set a sensible limit where the service is started:

touch config.ini
ulimit -n 50
python3 - <<'EOF'
import os, resource
for i in range(100):
    with open("config.ini") as f:      # closed at the end of the block
        f.read()
print("100 reads OK, open descriptors:", len(os.listdir("/proc/self/fd")))
print("soft/hard limit:", resource.getrlimit(resource.RLIMIT_NOFILE))
EOF
sleep 5 & pid=$!
echo "descriptors of pid $pid: $(ls /proc/$pid/fd | wc -l)"
prlimit --pid "$pid" --nofile | tail -n 1 | awk '{ print "limit for", $1 ":", $(NF-2), "soft,", $(NF-1), "hard" }'
kill $pid

100 reads OK, open descriptors: 4
soft/hard limit: (50, 50)
descriptors of pid 3921287: 3
limit for NOFILE: 50 soft, 50 hard

• Use with (Python), try-with-resources (Java), defer f.Close() (Go), and close HTTP responses.
• Monitor descriptor counts: ls /proc/<pid>/fd | wc -l or the process_open_fds metric.
• Set limits where the service starts: LimitNOFILE=65536 in the systemd unit, --ulimit nofile= in Docker.

Trade-offs

• Raising the limit is right for servers with many connections, but it only delays a leak.
• Very high limits make leaks take longer to notice; alert on growth instead.
• ulimit in a shell profile doesn't apply to systemd services; set it in the unit file.

Detect it in production

• Graph open descriptors per process; a steadily rising line is a leak.
• lsof -p <pid> | awk '{print $5}' | sort | uniq -c shows which kinds of descriptors dominate.
• Load tests that run for hours catch leaks that short tests miss.

Key points

• Every file and socket uses a descriptor, and processes have a limit.
• Fix leaks by closing resources deterministically; raising limits only buys time.
• Set limits in the service manager and monitor descriptor counts.

Further reading: getrlimit(2) · systemd.exec: LimitNOFILE