Text Manipulation — Slicing Logs With Small Tools
notes-api suddenly slows down and you suspect one client is hammering it. The answer is in a 200 MB access log. You could open it in an editor and scroll, or you could get the top five clients with one line of commands in a few seconds.
The rule
Small tools, one job each, joined by pipes.
Unix text tools each do one thing to a stream of lines. The pipe (a vertical bar in the shell) sends one tool's output into the next. The essential set:
| Tool | Its one job | Example |
|---|---|---|
grep |
keep lines that match | grep " 500 " access.log |
cut |
take fields by a delimiter | cut -d, -f2 users.csv |
awk |
take whitespace-separated fields, do small logic | awk '{print $1}' access.log |
sed |
find and replace | sed 's/http:/https:/g' links.txt |
sort |
order lines | sort -n for numbers, -r to reverse |
uniq -c |
collapse adjacent repeats and count them | |
wc -l |
count lines | |
head / tail |
first or last lines; tail -f follows a growing file |
A line of our access log looks like this:
203.0.113.9 - - [12/Mar/2026:10:15:02 +0000] "GET /notes/42 HTTP/1.1" 200 512The first field is the client IP, so this pipeline finds the busiest clients:
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -5The shortcut
Build the pipeline one stage at a time, looking at the output after each stage. Say the question as a chain of verbs: pick the IP, group equal ones, count them, biggest first, keep five. Each verb is one tool.
Pick the field
awk '{print $1}' access.logprints only the IP of every request.
The classic trap is uniq -c without sort before it. uniq only merges repeats that are next to each other, so an unsorted log gives the same IP many times with small counts.
Drills
Answer fast — trust the rule.
Why does the pipeline sort before running uniq -c?
What does sed 's/http:/https:/g' links.txt do?
Which command prints the second comma-separated field of each line in users.csv?
Which command counts the lines of access.log that contain 500?
Remember
- Small tools, one job each, joined by pipes.
- Shortcut: say the question as verbs, add one tool per verb, check after each stage.
- Trap:
uniqwithoutsortfirst.