Normal and peak - measuring the load you actually have

Every capacity argument starts with a number, and the number is almost always wrong. Someone opens a dashboard, sees 22% CPU, and concludes the cluster is a third full. What they read was one instant on one node, smoothed by whatever averaging the dashboard applies at that zoom level.

The previous lesson gave you the ceiling. This one gives you the input: what the load actually is, measured over a window long enough to contain the load that matters.

Two numbers, not one

A capacity baseline is a pair.

• Normal: what the system carries most of the time. Use the median or the mean over a representative window. It sets the cost of running the service.
• Peak: what the system carries when it is busiest. Use a high percentile - p95 or p99 of the sampled interval - not the single highest sample. It sets the capacity you must own.

You size against the peak and you budget against the normal. A plan that names only one of them is answering only half the question, and it is usually the half nobody was asking.

The window matters as much as the statistic. A number without “measured over the fortnight of 21 July to 3 August, business hours and overnight batch included” attached to it cannot be compared with next quarter’s number.

Where the history already is

You do not need a metrics stack to build a first baseline. On Debian, Ubuntu, RHEL and their derivatives, installing the sysstat package gives you sar, which writes a binary sample file per day under /var/log/sysstat/ (Debian family) or /var/log/sa/ (RHEL family). The default retention is seven to twenty-eight days depending on the distribution.

$ ls -1 /var/log/sysstat/ | head; grep -E '^(ENABLED|HISTORY)' /etc/sysstat/sysstat /etc/default/sysstat 2>/dev/null

sa05
sa06
sa07
sa08
sa09
sa10
sa11
/etc/sysstat/sysstat:HISTORY=7
/etc/default/sysstat:ENABLED="true"

sar with no arguments reads today’s file. -f reads a specific day, and -s and -e narrow to a time range within it.

$ sar -u -f /var/log/sysstat/sa10 -s 09:00:00 -e 18:00:00 | tail -5

04:50:01 PM     all     18.42      0.00      3.11      1.04      0.00     77.43
05:00:01 PM     all     21.07      0.00      3.44      0.92      0.00     74.57
05:10:01 PM     all     19.88      0.00      3.20      1.11      0.00     75.81
Average:        all     17.63      0.00      2.98      0.97      0.00     78.42

The averaging trap

The default collection interval on most distributions is ten minutes. Everything shorter than ten minutes has already been averaged away before you ever run sar, and then the daily Average: line averages the averages.

Work an example. A service handles a synchronous batch every weekday at 02:00 that pins the CPU for eleven minutes, and idles at roughly 8% the rest of the time.

Busy period:   11 min at 96%
Quiet period: 1429 min at 8%

Daily mean = (11 x 96 + 1429 x 8) / 1440
           = (1056 + 11432) / 1440
           = 12488 / 1440
           = 8.7%

The dashboard says 8.7%. The honest statement is “8% normally, 96% for eleven minutes every weekday night”. Size this host on 8.7% and the batch window becomes an incident the first time the input doubles - and it will fail at 02:00, which is the worst time for it to fail and the least likely time for anyone to notice the trend beforehand.

Extracting a percentile

sar prints averages, not percentiles. sadf -d converts the binary file to a semicolon-separated form you can push through awk or sort, which is enough for a p95 without any extra tooling.

$ sadf -d /var/log/sysstat/sa10 -- -u | awk -F';' 'NR>1 {print 100 - $NF}' | sort -n | awk '{v[NR]=$1} END {printf "n=%d  median=%.1f%%  p95=%.1f%%  max=%.1f%%\n", NR, v[int(NR*0.50)], v[int(NR*0.95)], v[NR]}'

n=144  median=9.4%  p95=61.8%  max=96.2%

Median 9.4%, p95 61.8%, max 96.2%. Those three numbers describe the host. The single Average: line did not.

Run the same extraction across every retained day and you have a baseline rather than a snapshot:

for f in /var/log/sysstat/sa[0-9][0-9]; do
  printf '%s ' "$f"
  sadf -d "$f" -- -u \
    | awk -F';' 'NR>1 {print 100 - $NF}' \
    | sort -n \
    | awk '{v[NR]=$1} END {printf "median=%.1f p95=%.1f max=%.1f\n", v[int(NR*0.50)], v[int(NR*0.95)], v[NR]}'
done

Peak-to-mean ratio

The ratio of peak to normal is the single most useful derived number in a capacity baseline, because it tells you how much of your hardware exists solely to absorb bursts.

peak-to-mean = p95 utilisation / median utilisation

For the host above: 61.8 / 9.4 = 6.6. Two thirds of a working day it is nearly idle; for the rest it needs almost seven times its typical draw.

Read the ratio like this:

A high ratio is not a fault, but it is a decision point. Buying hardware to sit idle 95% of the time is one legitimate answer. Spreading the burst - staggering cron, rate-limiting the batch, smoothing the queue - is usually the cheaper one, and it is invisible as an option until somebody computes the ratio.

Do not baseline one host

Take the baseline per role, across every member of the role, and look at the spread before you take an average of the averages.

$ for h in web01 web02 web03; do printf '%s ' "$h"; ssh "$h" 'sar -u -f /var/log/sysstat/sa10 | tail -1'; done

web01 Average:        all     17.63      0.00      2.98      0.97      0.00     78.42
web02 Average:        all     18.10      0.00      3.02      1.01      0.00     77.87
web03 Average:        all     41.55      0.00      6.71      2.30      0.00     49.44

The cluster’s capacity is bounded by its hottest member, not by its mean. An uneven distribution means the fleet average is describing a machine that does not exist.

Collect the other resources too

CPU is the easiest to measure and rarely the constraint. Take the same median/p95/max treatment for each of these:

• Memory: sar -r, watching kbavail rather than kbfree, since page cache is reclaimable
• Swap activity: sar -W - pswpin and pswpout above zero means memory is already the constraint
• Disk throughput and queueing: sar -d -p, and iostat -xz for r_await and w_await
• Filesystem space and inodes: df -h and df -i, sampled on a timer since sar does not record them
• Network: sar -n DEV for throughput and sar -n EDEV for errors and drops
• Run queue and load: sar -q LOAD
• Pressure stall: sar -q PSI on kernels that expose /proc/pressure

Whichever of those reaches its own ceiling first is the one that sets your capacity date. Baselining only CPU means discovering the real constraint during the incident.

Knowledge check