sed and awk in production · Linux

sed and awk are full programming languages specialised for stream text processing. Together with grep they cover virtually every ad-hoc log-triage task a sysadmin faces.

sed — line-oriented transformations

The basic sed command:

sed [-n] 'address{command}' file

address — which lines to act on (1, 5, 10,20, /regex/, $).
command — what to do (s substitute, d delete, p print, i insert, a append, c change, y transliterate).

The most common production pattern: substitution.

Read-only / Safesed substitute— s/from/to/ replaces the first match on each line. Add /g to replace all matches. The s/// delimiter can be any character: s#/var/log#/var/log/old# avoids escaping slashes.

$ echo 'Hello World' | sed 's/World/Linux/'

Hello Linux

Illustrative output

Read-only / Safesed delete— 2d deletes line 2. Combine with addresses for surgical edits: sed '/^#/d' deletes comment lines, sed -n '/start/,/end/p' prints a range.

$ printf 'a\nb\nc\n' | sed '2d'

a
c

Illustrative output

Read-only / Safesed range extraction— -n suppresses default printing; /start/,/end/p prints only the matched range. Useful for slicing out configuration blocks.

$ cat /etc/ssh/sshd_config | sed -n '/^Port/,/^$/p' | head -5

Port 22

#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::

Illustrative output

awk — field-oriented programming

awk reads input as records (lines by default) split into fields. A program is a series of pattern { action } clauses:

awk 'condition { action }' file

If condition matches, action runs. $1, $2, … are the fields; $0 is the whole line; NF is the field count; NR is the record number; FS is the field separator (default: whitespace).

Read-only / Safeawk UID 0 audit— -F: sets the field separator to colon. $3 == 0 matches UID 0 (root). The action prints a warning. Useful for auditing /etc/passwd for unexpected root-equivalent accounts.

$ awk -F: '$3 == 0 { print "UID 0 account:", $1 }' /etc/passwd

UID 0 account: root

Illustrative output

Read-only / Safeawk frequency count— For each line, increment a counter keyed by field 1 (the shell column of /etc/passwd). END runs after all input. Sort -rn gives the most common shells first.

$ awk '{ counts[$1]++ } END { for (u in counts) printf "%d %s
", counts[u], u }' /etc/passwd | sort -rn | head -5

46 /
1 /usr/sbin
1 /bin/sync
1 /bin/shutdown
1 /sbin

Illustrative output

Read-only / Safeawk numeric range— A common audit: list human (or system-but-not-service) accounts. UID 1000+ is the typical human range on Debian-family; UID < 1000 is system accounts. Adjust for your distribution.

$ awk -F: '$3 >= 1000 && $3 < 65534 { print $1, $3 }' /etc/passwd | head

nobody 65534
ssm-user 1001
deploy 1002

Illustrative output

awk vs the basic toolkit

When to reach for awk instead of grep + cut + sed:

Task	Reach for
Filter lines by content	grep
Extract a fixed column from delimited text	cut
Substitute on a per-line basis	sed
Multiple delimiters, escaped fields, or field-based decisions	awk
Per-record arithmetic (sums, averages, percentiles)	awk
Per-record printing driven by previous records	awk
Pretty-printing structured output	awk

A real production example: nginx access log triage

The combination of grep + awk + sort + uniq is the canonical nginx/Apache access-log diagnostic.

Read-only / Safetop 5xx URLs— Print the request URI (field 7) for every request whose status code (field 9) starts with 5. Then count and rank. This identifies the endpoints producing server errors.

$ awk '$9 ~ /^5/ { print $7 }' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head

...

Illustrative output

sed — line-oriented transformations

awk — field-oriented programming

awk vs the basic toolkit

A real production example: nginx access log triage

Knowledge check