sort, uniq and tr — Ordering, Counting and Changing Characters
"Which crop did we pick most often?" is one of the most common questions you can ask about data. It turns out to be a single line of shell: slice out a column, put it in order, count the repeats, and put the counts in order. tr joins them as a tool for tidying text one character at a time.
The rule
sort before uniq, always.
sort puts lines in order. By default it sorts alphabetically, and a few options change that:
| Option | Does |
|---|---|
-n |
numeric order, so 10 comes after 9 |
-r |
reverse the order |
-t, |
fields are separated by commas |
-k3 |
sort starting from field 3 |
-u |
keep only one of each identical line |
$ tail -n +2 harvest.csv | sort -t, -k3 -n | head -n 1
2026-06-01,basil,0.4,raviThat line finds the smallest pick: skip the header, sort by kilograms numerically, and keep the first line.
uniq removes duplicate lines, but only when they are next to each other. That's why it nearly always comes after sort. uniq -c puts a count in front of each line:
$ tail -n +2 harvest.csv | cut -d, -f2 | sort | uniq -c
2 basil
1 chilli
3 tomatotr (translate) changes or deletes characters. It only reads stdin, so you feed it through a pipe or with <:
$ echo tomato | tr a-z A-Z # TOMATO
$ echo "kg 3.5" | tr -s ' ' # squeeze repeated spaces: kg 3.5
$ tr -d '\r' < windows.csv > unix.csv # delete Windows carriage returns
$ tr ',' '\t' < harvest.csv # commas become tabsTwo tab-tidying cousins: expand turns tabs into spaces, and unexpand turns runs of spaces back into tabs.
The shortcut
"How many of each, most first?" always has the same four stages: slice, sort, count, sort again. That's cut, then sort, then uniq -c, then sort -rn.
Slice
tail -n +2 harvest.csvpiped intocut -d, -f2gives just the crop names.
The trap is running uniq -c on unsorted input. The crops arrive as tomato, basil, tomato, chilli, tomato, basil, and no two neighbours are the same, so every count comes out as 1.
Drills
Answer fast — trust the rule.
Why do you almost always run sort before uniq?
A file holds the lines 9, 10 and 100. What does plain sort output?
Which command turns basil into BASIL?
How many lines does the four-stage pipeline print, from tail -n +2 harvest.csv through cut -d, -f4, sort and uniq -c?
Remember
- sort before uniq, always, because uniq only merges repeats that sit next to each other.
- "How many of each" means slice, sort, count, sort again:
cut,sort,uniq -c,sort -rn. - Use
sort -nfor numbers, and remember thattronly reads stdin.