Why is a while-read loop with $(echo "$line" | cut ...) slow?

Each iteration starts several new processes; process creation dominates, so run time grows with the number of lines.

How would you count requests per status code in an access log?

In one pass: awk '{ c[$9]++ } END { for (s in c) print s, c[s] }' access.log, or cut -d' ' -f9 access.log | sort | uniq -c.

When should a shell script become a Python or Go program?

When it needs complex logic, data structures, error handling or tests, or processes large data with non-trivial rules.

A Shell Script That Takes Hours — Linux and Bash for Developers

Scenario

A nightly script summarises an access log. It took a minute when the site was small; now it runs for hours and overlaps with the next night's run.

What you see in production

High CPU in short-lived processes (grep, cut, awk) started thousands of times per second.
Run time grows faster than the input.
The script is a while read loop calling external commands for every line.

The naive code

while read -r line; do
  status=$(echo "$line" | cut -d' ' -f9)
  path=$(echo "$line" | awk '{ print $7 }')
  if [ "$status" = "500" ]; then echo "$path" >> errors.txt; fi
done < access.log

Reproduce it

One process per line versus one pass

for i in $(seq 1 2000); do
  status=$(( i % 50 == 0 ? 500 : 200 ))
  echo "10.0.0.$(( i % 9 )) - - [15/Jan/2026:10:00:00 +0000] \"GET /api/item/$(( i % 20 )) HTTP/1.1\" $status 512"
done > access.log
start=$(date +%s%N)
: > slow.txt
while read -r line; do
  status=$(echo "$line" | cut -d' ' -f9)
  path=$(echo "$line" | awk '{ print $7 }')
  [ "$status" = "500" ] && echo "$path" >> slow.txt
done < access.log
slow_ms=$(( ($(date +%s%N) - start) / 1000000 ))
start=$(date +%s%N)
awk '$9 == 500 { print $7 }' access.log > fast.txt
fast_ms=$(( ($(date +%s%N) - start) / 1000000 ))
cmp -s slow.txt fast.txt && echo "same result: $(wc -l < fast.txt) error lines"
(( slow_ms > 20 * (fast_ms + 1) )) && echo "the per-line loop was more than 20x slower"

Output

same result: 40 error lines
the per-line loop was more than 20x slower

Run this on your computer — see “Get Started” for setup.

Root cause

Each $(...) with a pipeline forks new processes: two or three per line here. Starting a process costs around a millisecond, so a million lines means hours of process creation. The actual text processing is trivial.

The fix

Let one program read the whole file: awk, sort, uniq and grep stream through data in a single process:

A full summary in one pipeline

for i in $(seq 1 2000); do
  status=$(( i % 50 == 0 ? 500 : 200 ))
  echo "10.0.0.$(( i % 9 )) - - [15/Jan/2026:10:00:00 +0000] \"GET /api/item/$(( i % 20 )) HTTP/1.1\" $status 512"
done > access.log
awk '{ total++; by_status[$9]++ } $9 >= 500 { errors[$7]++ }
     END {
       printf "requests: %d\n", total
       for (s in by_status) printf "status %s: %d\n", s, by_status[s]
       for (p in errors) printf "errors on %s: %d\n", p, errors[p]
     }' access.log | sort

Output

errors on /api/item/0: 20
errors on /api/item/10: 20
requests: 2000
status 200: 1960
status 500: 40

Run this on your computer — see “Get Started” for setup.

Replace per-line echo | cut/awk with one awk program, or Bash builtins (read -r a b c, ${var#...}) when you must loop.
Use sort | uniq -c for counting, grep -c for matching counts.
When logic gets complex, switch to Python or Go, which also process lines in one process.

Trade-offs

Long awk programs get hard to read; move them into a file with comments, or switch languages.
Bash builtins avoid forks but are still slow per line compared with awk.
For huge logs, LC_ALL=C speeds up sort and grep by skipping locale-aware comparisons.

Detect it in production

Time scripts in CI with realistic input sizes.
strace -c -f shows thousands of clone/execve calls for fork-heavy scripts.
Watch for scheduled jobs whose run time trends upward.

Key points

Every $(...) or pipeline in a loop starts new processes; that's the cost.
Process whole files in one awk/sort/grep pass.
Move complex logic to a real programming language.

Further reading: gawk(1) · Bash tutorial: awk