The Disk Is Full but du Finds Nothing

Scenario

An alert fires: the root filesystem is 98% full. You delete a 2 GB debug log with rm, but df still shows 98%. du -sh /* adds up to far less than df reports. Someone suggests rebooting the server.

What you see in production

• df -h and du -sh disagree by gigabytes.
• Deleting files doesn't free space.
• A long-running process (an app server, a log shipper) was writing to the file you removed.

The naive code

# "Clean up" a huge log while the app is still writing to it
rm /var/log/app/debug.log
df -h /var/log      # still full

Reproduce it

A background process keeps a file open; after rm the name is gone, but the data stays on disk until the last open file descriptor is closed. /proc/<pid>/fd shows it as (deleted):

python3 - <<'EOF' &
import time
f = open("debug.log", "w")
f.write("x" * 50_000_000)          # 50 MB
f.flush()
time.sleep(5)                      # keep the file open, like a server would
EOF
pid=$!
sleep 1
ls -lh debug.log | awk '{ print $5, $9 }'
rm debug.log
echo "after rm, files here: $(ls | wc -l)"
ls -l /proc/$pid/fd | grep -o 'debug.log (deleted)'
size=$(stat -L -c %s /proc/$pid/fd/3)
echo "still held open by pid: $(( size / 1000000 )) MB"
kill $pid; wait $pid 2>/dev/null || true
echo "process stopped: the space is released now"

48M debug.log
after rm, files here: 0
debug.log (deleted)
still held open by pid: 50 MB
process stopped: the space is released now

Root cause

On Linux a file is deleted when its last name and last open descriptor are gone. rm only removes the name (the directory entry). df counts blocks in use on the filesystem; du walks the directory tree and can't see files without names.

The fix

Find the processes holding deleted files, then release the space properly:

python3 -c 'import time; f = open("big.log", "w"); f.write("y" * 20_000_000); f.flush(); time.sleep(5)' &
pid=$!
sleep 1
rm big.log
# Which descriptors point at deleted files? On a server, `lsof +L1` lists them for every process.
for fd in /proc/$pid/fd/*; do
  target=$(readlink "$fd")
  if [[ $target == *"(deleted)" ]]; then
    held=$fd
    echo "pid $pid fd ${fd##*/} -> ${target##*/}, $(stat -L -c %s "$fd") bytes"
  fi
done
# Truncate through the open descriptor: the space is freed and the process keeps running
: > "$held"
echo "after truncate: $(stat -L -c %s "$held") bytes"
kill $pid; wait $pid 2>/dev/null || true

pid 3921152 fd 3 -> big.log (deleted), 20000000 bytes
after truncate: 0 bytes

1. Find the holders: lsof +L1 (or the /proc loop above).
2. Prefer truncating instead of deleting a log a process writes to: : > app.log or truncate -s 0 app.log.
3. Or make the process reopen its files (a reload signal such as SIGHUP/SIGUSR1, or a restart).
4. Prevent it: rotate logs with logrotate (next lesson) and alert on disk usage trends, not just thresholds.

Trade-offs

• Truncating loses the log's content; copy what you need first.
• Restarting the process is simplest but may drop connections; a reload signal is gentler if the app supports it.
• Rebooting "works" but hides the cause and takes the service down.

Detect it in production

• Alert when df and du diverge significantly, and on disk growth rate.
• Run lsof +L1 (or find /proc/*/fd -lname '*(deleted)') during disk incidents.
• In containers, deleted-but-open files count against the writable layer or volume too.

Key points

• rm removes a name; space is freed only when no process has the file open.
• df and du disagreeing usually means deleted-but-open files.
• Truncate logs instead of deleting them, and rotate them properly.

Further reading: lsof(8) · proc(5): /proc/pid/fd · Bash tutorial: files