Why does a process keep writing to app.log.1 after you rename app.log?

Its open file descriptor points to the inode, not the name; renaming doesn't change which file it writes to.

What's the trade-off of logrotate's copytruncate?

It works with apps that can't reopen logs, but lines written between the copy and the truncate can be lost.

What's the simplest logging approach for services on modern platforms?

Write logs to stdout/stderr and let journald, Docker or Kubernetes collect and rotate them.

Logs Rotate but the Disk Keeps Filling — Linux and Bash for Developers

Scenario

A team adds a nightly job that renames app.log to app.log.1 and compresses old copies. A week later the disk is full again: app.log is tiny, but app.log.1 keeps growing, and the compressed copies are empty.

What you see in production

The "rotated" file keeps growing after rotation.
The new app.log stays empty or is never created.
Compressed archives are created from files still being written to.

The naive code

mv app.log app.log.1
gzip app.log.1
touch app.log

Reproduce it

A writer keeps an open descriptor; renaming the file doesn't change which inode the descriptor points at:

Renaming a log the app is writing to

python3 - <<'EOF' &
import time
log = open("app.log", "a", buffering=1)
for i in range(6):
    log.write(f"request {i}\n")
    time.sleep(0.3)
EOF
pid=$!
sleep 0.5
mv app.log app.log.1
touch app.log
wait $pid
echo "app.log:   $(wc -l < app.log) lines"
echo "app.log.1: $(wc -l < app.log.1) lines (the app kept writing here)"

Output

app.log:   0 lines
app.log.1: 6 lines (the app kept writing here)

Run this on your computer — see “Get Started” for setup.

Root cause

An open file descriptor refers to an inode, not a name. mv changes the name; the process keeps writing to the same inode, now called app.log.1. Until the process reopens its log file, the new app.log is never written.

The fix

Either tell the process to reopen its logs after rotating (best), or copy and truncate in place (copytruncate) for apps that can't reopen. logrotate does both:

logrotate with copytruncate

mkdir -p logs
python3 - <<'EOF' &
import time
log = open("logs/app.log", "a", buffering=1)
for i in range(8):
    log.write(f"request {i}\n")
    time.sleep(0.25)
EOF
pid=$!
cat > logrotate.conf <<EOF
$PWD/logs/app.log {
    rotate 7
    daily
    missingok
    notifempty
    copytruncate
    compress
    delaycompress
}
EOF
sleep 0.9
/usr/sbin/logrotate --force --state "$PWD/logrotate.state" logrotate.conf
wait $pid
ls logs
echo "app.log.1: $(wc -l < logs/app.log.1) lines, app.log: $(wc -l < logs/app.log) lines (new writes land here)"

Output

app.log
app.log.1
app.log.1: 4 lines, app.log: 4 lines (new writes land here)

Run this on your computer — see “Get Started” for setup.

For apps that support it, use postrotate with a reload signal (for example systemctl reload nginx or kill -USR1) instead of copytruncate.
Better still for services: log to stdout, and let journald, Docker or Kubernetes handle storage and rotation.

Trade-offs

copytruncate can lose lines written between the copy and the truncate.
Reopen-on-signal is lossless but needs app support.
Shipping logs to a central system (Loki, Elasticsearch, CloudWatch) reduces local retention needs.

Detect it in production

Alert on disk growth rate and on files under /var/log larger than expected.
Check rotated files' timestamps: a .1 file with a recent modification time means the app is still writing to it.
logrotate --debug /etc/logrotate.d/app shows what would happen without rotating.

Key points

Processes write to inodes; renaming a file doesn't redirect their writes.
Rotate with logrotate: reopen via postrotate, or copytruncate if you must.
For services, log to stdout and let the platform rotate.

Further reading: logrotate(8) · systemd journald