Sign in Get started

Linux and Bash for Developers · Capstone

Capstone: Investigate a Slow Server and Write the Runbook

Work through a realistic Linux incident from first alert to root cause, and turn it into a runbook and preventive changes.

Create a free account to save this page, take notes and track your progress. Sign in

Scenario

At 02:10 the on-call engineer gets three alerts for the same Linux VM that runs an API (systemd service api, Nginx in front) and a nightly report job (cron, 02:00): API latency is ten times normal, disk usage is 95%, and the report job hasn't finished. SSH works but feels slow.

Requirements

  • Restore normal latency without losing data.
  • Find the root cause, not just the symptoms.
  • Make sure it doesn't happen again, and that the next engineer can handle it faster.

Constraints

  • Production server, no reboot without approval.
  • You have sudo, journalctl, ss, lsof, ps, top, df, du, dmesg.

Deliverables

  1. First five minutes: what you look at, in which order.
  2. Investigation: commands and what each tells you.
  3. Likely root cause and immediate fix.
  4. Preventive changes.
  5. Runbook.

Hints

  • Two of the alerts may have the same cause.
  • Check what changed recently, including scheduled jobs.

Draft each deliverable below before revealing the model answer and rubric.

Further reading: Brendan Gregg: USE method · Google SRE book: effective troubleshooting

Deliverables

What do you look at in the first five minutes?

Model answer

  • uptime (load), top/htop (CPU, memory, which processes), free -h, df -h and df -i (space and inodes).
  • journalctl -u api --since -30min and Nginx error logs for errors and timeouts.
  • systemctl list-timers / crontab: the report job started at 02:00, just before the alerts.
  • dmesg -T | tail for OOM kills and disk errors.

Rubric

Look for a quick, systematic sweep of load, memory, disk, logs and recent changes (USE method), noticing the timing of the cron job.

Which commands do you run to investigate, and what do they show?

Model answer

  • ps -eo pid,etime,pcpu,pmem,args --sort=-pcpu | head: the report job (a bash loop calling external commands per line) uses most CPU.
  • du -xh /var --max-depth=2 | sort -h | tail and lsof +L1: a debug log was deleted while the API still holds it open; the report writes a huge temporary file.
  • iostat -x 1 / vmstat 1: high I/O wait from the report's writes slows the API.
  • ls /proc/$(pidof api)/fd | wc -l: descriptor count is normal (rules out a leak).

Rubric

Strong answers form hypotheses and test them with specific commands, ruling causes in and out.

What's the likely root cause and the immediate fix?

Model answer

  • The nightly report grew with the data: a fork-per-line Bash loop pegs the CPU and writes a large temp file, while a deleted-but-open debug log already held space. Together they fill the disk and starve the API of CPU and I/O.
  • Immediate: stop the report job (gracefully), truncate the deleted log through /proc/<pid>/fd/<n> or reload the API's logging, remove the job's temp file, confirm latency and disk recover.

Rubric

Look for a root cause that explains all three alerts and an immediate fix that's safe (no reboot, no data loss).

Which preventive changes do you make?

Model answer

  • Rewrite the report as a single awk/Python pass, stream data, and run it with nice/ionice or a systemd unit with CPUQuota=/MemoryMax=.
  • Put temp files on a separate volume or clean them up with trap; logrotate for the API logs (or log to the journal).
  • Alerts on disk growth rate and on job duration; heartbeat monitoring for the job.
  • Move the job to a systemd timer for logs and resource limits.

Rubric

Strong answers fix the job's design, isolate its resources, and improve monitoring so the next occurrence is caught earlier.

What goes into the runbook?

Model answer

  • Symptoms and alerts that trigger it; a checklist of commands (load, memory, disk, deleted files, top processes, recent jobs) with what "normal" looks like.
  • Safe actions in order (stop job, truncate held files, reload logging) and actions that need approval (restart API, reboot).
  • How to verify recovery, who to inform, and links to dashboards and the postmortem.

Rubric

Look for a runbook another engineer could follow at 2 a.m.: symptoms, diagnostics with expected values, safe ordered actions, and verification.

Open in the app to mark lessons complete, take notes and track your progress.

Was this page helpful?