Sign in Get started

DevOps Tutorial · Linux and the Shell · Lesson 9 of 25

Text Processing: grep, sed, awk

Search and summarise logs and command output with grep, sort, uniq, cut, sed and awk.

Create a free account to save this page, take notes and track your progress. Sign in

The Unix toolbox

Small tools connected with pipes (|) answer most "what's in these logs?" questions:

Tool Does Example
grep lines matching a pattern grep -i error app.log, grep -v healthz
sort, uniq -c sort, count duplicates `sort
cut pick fields cut -d' ' -f1
sed find and replace sed 's/password=[^ ]*/password=***/'
awk process columns, compute awk '{sum += $NF} END {print sum}'
wc -l count lines
Find errors and top clients
cat > access.log <<'LOG'
203.0.113.5 GET /api/orders 200 120
198.51.100.7 GET /healthz 200 2
203.0.113.5 POST /api/orders 500 900
192.0.2.44 GET /api/products 200 80
203.0.113.5 GET /api/orders 200 95
198.51.100.7 GET /api/orders 502 30
LOG
echo "errors (5xx):"
grep -E ' 5[0-9]{2} ' access.log
echo "requests per client:"
cut -d' ' -f1 access.log | sort | uniq -c | sort -rn

Output

errors (5xx):
203.0.113.5 POST /api/orders 500 900
198.51.100.7 GET /api/orders 502 30
requests per client:
      3 203.0.113.5
      2 198.51.100.7
      1 192.0.2.44

Run this on your computer — see “Get Started” for setup.

Summarise with awk

Average latency per path with awk
cat > access.log <<'LOG'
203.0.113.5 GET /api/orders 200 120
198.51.100.7 GET /healthz 200 2
203.0.113.5 POST /api/orders 500 900
192.0.2.44 GET /api/products 200 80
203.0.113.5 GET /api/orders 200 95
LOG
grep -v /healthz access.log | awk '
  { count[$3]++; total[$3] += $5 }
  END { for (path in count) printf "%-14s %d requests, avg %.0f ms\n", path, count[path], total[path] / count[path] }
' | sort

Output

/api/orders    3 requests, avg 372 ms
/api/products  1 requests, avg 80 ms

Run this on your computer — see “Get Started” for setup.

Redact with sed

Redact a secret before sharing a log line
echo 'connect user=app password=s3cret host=db.internal' | sed -E 's/password=[^ ]+/password=***/'

Output

connect user=app password=*** host=db.internal

Run this on your computer — see “Get Started” for setup.

Test yourself

Which pipeline counts how many times each line occurs, most frequent first?

  1. sort | uniq -c | sort -rn
  2. uniq | sort
  3. grep -c | sort
  4. cut | wc -l
Show answer

Answer: sort | uniq -c | sort -rn

uniq only collapses adjacent duplicates, so sort first; -c counts and sort -rn orders by count.

Was this page helpful?