sar -n - network history and the counters nobody collected · Linux

Live network tools answer “what is happening now”. Every incident review asks “what was happening at 02:14 last Thursday”, and by then the counters have rolled over and the connection is closed.

sysstat answers that question, if it was collecting the right things. The second half of that sentence is where most of this lesson lives.

Reading a past window

Data files live in /var/log/sysstat (/var/log/sa on RHEL), one per day, named saNN where NN is the day of the month:

ls /var/log/sysstat/

sa03  sa04  sa05  sa06  sa07  sa08  sa09

Point sar at the file for the day and bound the window:

S_TIME_FORMAT=ISO sar -n DEV --iface=ens18 -f /var/log/sysstat/sa03 \
  -s 12:00:00 -e 12:40:00

Linux 7.0.0-27-generic (orchgrid) 	2026-08-03 	_x86_64_	(4 CPU)

12:00:37        IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s   %ifutil
12:10:37        ens18     15.72     10.93      1.75     21.96      0.00      0.00      0.00      0.00
12:20:19        ens18     20.57     13.52      2.34     34.44      0.00      0.00      0.00      0.00
12:30:00        ens18     28.45     18.92      2.96     58.05      0.00      0.00      0.00      0.00
Average:        ens18     21.52     14.42      2.34     37.97      0.00      0.00      0.00      0.00

S_TIME_FORMAT=ISO is worth making a habit. Without it sar uses the locale time format, which on many systems means a 12-hour clock with a separate AM/PM field - so every column position shifts by one and any awk you write against the output silently reads the wrong field.

--iface= is not optional in practice. A host running containers has a veth per container plus bridges, and without the filter sar -n DEV prints dozens of interfaces per interval and the one you care about scrolls away:

ls /sys/class/net/

br-034ca77f5100  docker0  ens18  lo  veth429b399  veth8bac050  vethb7894d8

Which keyword answers which question

sar -n DEV     # throughput and packet rates per interface
sar -n EDEV    # errors and drops per interface
sar -n SOCK    # sockets in use, including TIME_WAIT
sar -n TCP     # connection rate and segment rate
sar -n ETCP    # retransmits, resets, checksum errors
sar -n UDP     # datagrams and receive errors

EDEV is the one to reach for when something was dropping:

sar -n EDEV --iface=ens18 -f /var/log/sysstat/sa03

Its columns are rxerr/s, txerr/s, coll/s, rxdrop/s, txdrop/s, txcarr/s, rxfram/s, rxfifo/s, txfifo/s. rxdrop/s and txdrop/s are drops for lack of buffer space in the kernel; rxfifo/s is a NIC ring overrun. They point at different fixes, which is why the split matters.

ETCP carries retrseg/s, the retransmission rate. Combined with oseg/s from TCP it gives the retransmit ratio over a past window - the number the previous lesson showed dominates throughput on a long path:

sar -n TCP,ETCP -f /var/log/sysstat/sa03 -s 02:00:00 -e 02:30:00

The counters you need may never have been collected

sadc does not record everything by default. Ask for TCP history on a host where it was not enabled and you get this:

Requested activities not available in file /var/log/sysstat/sa03

That message is easy to read as “nothing happened” or “the file is corrupt”. It means neither. It means the activity was never collected, and the data does not exist and cannot be recovered.

The man sar entry says so for each keyword - “TCPv4 statistics depend on sadc’s option -S SNMP to be collected”. Check what this host records:

grep -E '^(SADC_OPTIONS|HISTORY|COMPRESSAFTER)' /etc/sysstat/sysstat

HISTORY=7
COMPRESSAFTER=10
SADC_OPTIONS="-S DISK"

-S DISK only. So sar -n DEV, EDEV and SOCK work, and TCP, ETCP, UDP and IPv6 have no history at all - on a host where the interesting question is almost always about retransmits.

Fix it before you need it:

# /etc/sysstat/sysstat
SADC_OPTIONS="-S DISK,SNMP,IPV6"
HISTORY=31

sudo systemctl restart sysstat-collect.timer
sar -n TCP 1 1

HISTORY=7 is the other half of the trap: seven days of retention means a Monday incident reviewed a fortnight later has no data. Thirty-one days costs a few megabytes and is the difference between a review with evidence and one without.

Utilisation against link speed

%ifutil is computed from the interface speed the kernel reports. That works on a physical NIC and does not work on anything virtual:

for i in /sys/class/net/*; do
  printf '%-20s %s\n' "$(basename "$i")" "$(cat "$i/speed" 2>/dev/null || echo n/a)"
done

ens18                10000
docker0              -1
br-034ca77f5100      -1
lo                   n/a

-1 means the kernel has no speed for that interface, so %ifutil on it is meaningless - and it will report 0.00 rather than an error, which is the dangerous part. Only trust %ifutil on an interface whose speed file holds a real number.

For everything else, compute it. sar reports kB/s; the link speed is in Mbit/s:

S_TIME_FORMAT=ISO sar -n DEV --iface=ens18 -f /var/log/sysstat/sa03 \
  -s 12:00:00 -e 12:40:00 |
awk -v speed_mbit=10000 '
  $2 == "ens18" && $1 != "Average:" {
    rx_mbit = $5 * 8 / 1000
    tx_mbit = $6 * 8 / 1000
    printf "%s  rx %7.2f Mbit/s (%5.2f%%)  tx %7.2f Mbit/s (%5.2f%%)\n", \
           $1, rx_mbit, 100 * rx_mbit / speed_mbit, \
               tx_mbit, 100 * tx_mbit / speed_mbit
  }'

12:10:37  rx    0.01 Mbit/s ( 0.00%)  tx    0.18 Mbit/s ( 0.00%)
12:20:19  rx    0.02 Mbit/s ( 0.00%)  tx    0.28 Mbit/s ( 0.00%)
12:30:00  rx    0.02 Mbit/s ( 0.00%)  tx    0.46 Mbit/s ( 0.00%)

For a full-duplex link, utilisation is the larger of the two directions, not their sum - which is how sar defines %ifutil too.

The interval hides the event

systemctl cat sysstat-collect.timer | grep OnCalendar

OnCalendar=*:00/10

Every ten minutes. Each row of sar -n DEV is therefore an average over roughly 600 seconds, and a saturation event shorter than that is divided by 600 before you see it.

Concretely: a backup job saturates a 10 Gb link for 30 seconds. Averaged over ten minutes, that appears as 5% utilisation. The graph is flat. Meanwhile every request crossing that link during those 30 seconds timed out, and the incident report says “the network showed no anomaly”.

Three ways to deal with it:

Collect more often where it matters. A drop-in raises the rate without editing the vendor unit:

# /etc/systemd/system/sysstat-collect.timer.d/override.conf
[Timer]
OnCalendar=
OnCalendar=*:00/01

The empty OnCalendar= is required: it clears the vendor value before the new one is added, and omitting it leaves both active. One-minute collection multiplies the data volume by ten, so raise HISTORY deliberately or accept a shorter window.

Read EDEV rather than DEV. Errors and drops are counters, not rates - a 30-second burst still leaves its drops in the average, whereas throughput is diluted. rxdrop/s of 0.4 averaged over ten minutes means 240 packets were dropped in that window, which is a real finding even though the throughput graph is flat.

Keep a live capture running during known-risky windows. sar -n DEV 1 written to a file during a change window costs nothing and has the resolution the archive does not.

Get the data out with sadf rather than parsing sar output

sar output is formatted for humans and its column layout shifts between keywords and versions. sadf reads the same binary files and emits machine formats:

# CSV-like, one record per line, easy to load anywhere
sadf -d /var/log/sysstat/sa03 -- -n DEV --iface=ens18

# JSON
sadf -j /var/log/sysstat/sa03 -- -n DEV --iface=ens18

Everything after -- is passed to sar, so any selection that works there works here. Use this whenever the output is going into a script, a spreadsheet or an incident document; save the plain sar invocation for reading at a terminal.

An incident walkthrough

Reports of timeouts between 02:10 and 02:20 on the 3rd:

DAY=/var/log/sysstat/sa03

# Was the link busy?
sar -n DEV --iface=ens18 -f "$DAY" -s 02:00:00 -e 02:30:00

# Was anything dropped?
sar -n EDEV --iface=ens18 -f "$DAY" -s 02:00:00 -e 02:30:00

# Were sockets exhausted or piling up in TIME_WAIT?
sar -n SOCK -f "$DAY" -s 02:00:00 -e 02:30:00

# Was the host itself busy, or waiting on I/O?
sar -u -f "$DAY" -s 02:00:00 -e 02:30:00
sar -q -f "$DAY" -s 02:00:00 -e 02:30:00

Run all five before forming a hypothesis. The commonest outcome is that the network was quiet and the run queue was not, and the “network timeout” was a host that could not schedule the thread to read the socket. Historical data is worth having precisely because it lets you rule things out - and a negative result from the archive is evidence, whereas a missing activity is not.