No Space Left on Device

Scenario

Example team: a fleet of Docker hosts (and later Kubernetes nodes) runs about 20 services. One service logs every request body at DEBUG level, enabled during an investigation months ago. Another writes its own log file inside the container. CI builds also leave old images behind.

What you see in production

On a Sunday, one node after another reports "No space left on device". Containers can't start, the database's write-ahead log can't be written, Kubernetes marks nodes with DiskPressure and evicts pods, which reschedule onto other nodes that are also nearly full. The cascade takes down half the cluster.

The naive code

/etc/docker/daemon.json     (no log-driver options: json-file logs grow without limit)
LOG_LEVEL=DEBUG             (left on after an investigation)
app writes /app/logs/app.log inside the container's writable layer

Reproduce it

Disk usage grows steadily until it doesn't fit; linear growth makes it predictable, if anyone is looking:

DISK_GB = 100
base_gb = 38                             # OS, images, data
growth_gb_per_day = {"app logs (DEBUG)": 2.4, "container json logs": 0.9, "old images": 0.6}

total_rate = sum(growth_gb_per_day.values())
for day in range(0, 17, 4):
    used = base_gb + total_rate * day
    flag = "  <- 80% alert" if used >= 80 and used - total_rate * 4 < 80 else ""
    flag = "  <- FULL" if used >= DISK_GB else flag
    print(f"day {day:2}: {min(used, DISK_GB):5.1f} GB used ({min(used, DISK_GB) / DISK_GB:.0%}){flag}")
print(f"days until full: {(DISK_GB - base_gb) / total_rate:.1f}")

rotated = {"app logs (DEBUG)": 0, "container json logs": 0, "old images": 0}   # stdout + rotation + GC: bounded
print("with rotation and GC, growth per day:", sum(rotated.values()), "GB")

day  0:  38.0 GB used (38%)
day  4:  53.6 GB used (54%)
day  8:  69.2 GB used (69%)
day 12:  84.8 GB used (85%)  <- 80% alert
day 16: 100.0 GB used (100%)  <- FULL
days until full: 15.9
with rotation and GC, growth per day: 0 GB

Root cause

Nothing bounded the growth. Docker's default json-file log driver keeps container stdout forever unless max-size/max-file are set; files written inside a container grow its writable layer; unused images accumulate; and a debug log level multiplied the volume. Disk alerts, if any, fired at 95%, too late for a weekend.

The fix

1. Log to stdout/stderr (Twelve-Factor) and let the platform collect and ship logs; never write log files inside containers.
2. Rotate at the node level: Docker log-opts (max-size, max-file); Kubernetes kubelet containerLogMaxSize / containerLogMaxFiles.
3. Garbage-collect images: kubelet image GC thresholds; docker image prune on CI runners and plain Docker hosts.
4. Control volume: INFO by default, temporary debug logging with an expiry, sampling for high-volume logs.
5. Alert early and predictively: at 80%, and on "full within 24 hours" based on growth rate.

{
  "log-driver": "json-file",
  "log-opts": { "max-size": "50m", "max-file": "5" }
}

Trade-offs

• Rotation discards old local logs; ship them to central storage first if you need them.
• Log sampling saves cost but can hide rare events; never sample errors and audit logs.
• Bigger disks only postpone the problem; bounded growth fixes it.

Detect it in production

• Node disk usage alerts at 80% and predictive alerts (predict_linear in Prometheus) for "full in 24 h".
• Kubernetes DiskPressure node conditions and eviction events.
• Log volume per service per day, with owners notified on sudden increases.

Key points

• Logs and images grow without limit unless something bounds them.
• Log to stdout, rotate at the node, garbage-collect images, keep log levels sane.
• Alert at 80% and on growth rate, not at 95%.