Measure, change one thing, measure again - the tuning discipline · Linux

The tools in this part tell you what a host is doing. This lesson is about the discipline around them: which framework to reach for, how much measurement is enough to justify a change, and how to make the change in a way that somebody can undo in six months without knowing why you made it.

Most bad performance work is not caused by using the wrong tool. It is caused by changing three things at once, measuring once, and declaring victory.

USE for resources, RED for services

The USE method - utilisation, saturation, errors for every resource - is covered at the start of this part. It answers “what on this host is the bottleneck”. It is the right frame when you own the host and the question is about hardware or kernel resources.

RED is the complement, and it applies to a service rather than a resource:

Signal	Question	Typical source
Rate	How many requests per second is it handling?	Access log, application metric
Errors	How many of those failed?	Status codes, exception counters
Duration	How long did each one take, as a distribution?	Latency histogram, access log timing field

The two frameworks answer different questions and you usually need both. USE tells you the disk is saturated. RED tells you whether anybody noticed. A host with %util pinned at 100% and p99 latency flat has no incident; a host with plenty of idle CPU and p99 latency doubled has one.

The practical rule: start from RED if a user complained, and from USE if a monitor fired. A complaint is about a service, so begin at the service and work down into resources. An alert on a resource is about the host, so begin at the resource and work up to whether it matters.

One measurement is not a measurement

The single most common analytical error in tuning is running the workload once before a change and once after, seeing a difference, and attributing it to the change. Systems are noisy: page cache state, CPU frequency, a neighbouring VM, a backup job, and the scheduler all move results run to run.

Collect a distribution instead. This harness works for anything you can invoke from a shell:

#!/bin/bash
# Run a command N times and report the latency distribution in ms.
set -uo pipefail

N=${N:-30}
for _ in $(seq 1 "$N"); do
  start=$(date +%s%N)
  curl -s -o /dev/null "http://127.0.0.1:8080/health"
  end=$(date +%s%N)
  echo $(( (end - start) / 1000000 ))
done | sort -n | awk '{ a[NR] = $1 } END {
  printf "n=%d  min=%d  p50=%d  p95=%d  max=%d\n", \
         NR, a[1], a[int(NR*0.5)+1], a[int(NR*0.95)+1], a[NR]
}'

n=30  min=4  p50=6  p95=19  max=41

Now the “before” is five numbers rather than one, and the spread between p50 and max tells you how much noise you are working against. A change that moves p50 from 6 ms to 5 ms inside that spread has not been demonstrated to do anything.

Discard the warm-up, and say how many you discarded

The first few iterations of almost any measurement are unrepresentative: caches are cold, JITs have not compiled, connection pools are empty, and the CPU has not yet ramped its frequency. Including them drags the mean and inflates the maximum.

Run a handful of iterations and throw them away before the measured run:

for _ in $(seq 1 5); do curl -s -o /dev/null "http://127.0.0.1:8080/health"; done

Then take the real samples. Write down how many you discarded, because “we discarded 5 warm-up iterations” is the difference between a repeatable result and one nobody else can reproduce.

The warm-up is also the measurement, sometimes. If your service is restarted twenty times a day, cold-cache latency is user latency, and discarding it hides the thing you should be fixing.

Change exactly one thing

Once you have a baseline distribution, change one variable. Not one area - one variable.

BAD:  raised vm.dirty_ratio, switched the scheduler to none,
      and enabled write-back caching, then re-measured.
      p95 improved 30%. Which one did it?

GOOD: raised vm.dirty_ratio from 20 to 40. Re-measured.
      p95 unchanged. Reverted.
      Switched scheduler from mq-deadline to none. Re-measured.
      p95 improved 28%. Kept.

The good version costs three times as much wall-clock time and is the only one that produces knowledge. The bad version leaves you with three permanent changes, two of which do nothing, and one of which you will not be able to identify when it turns out to hurt a different workload next quarter.

Keep a control

Measure something you did not change, over the same window. If your control moves too, the difference is environmental and your change is not responsible.

# The control: a second, untouched host, measured in the same window
sar -u 1 60 > /tmp/control-cpu.txt

The cheapest control is a second host in the same role that you leave alone. The second cheapest is the same host measured immediately before and after with the change applied and reverted twice - A, B, A, B - which separates a real effect from a drift.

Record the change so it can be undone

A tuning change is only complete when somebody who was not there can find it and reverse it.

# Runtime only. A reboot undoes this. Exactly what you want while testing.
sudo sysctl -w vm.dirty_ratio=40

Runtime-only is the right first step: it is trivially reversible and it disappears on reboot if you walk away. Only once the change is demonstrated does it become permanent, and when it does, it goes into a file with a comment:

# /etc/sysctl.d/80-writeback.conf
# Raised from the default 20 on 2026-08-11.
# Reason: p95 write latency on the ingest path, ticket OPS-4412.
# Evidence: 30-sample fio randwrite, p95 41ms -> 22ms, reproduced twice.
# Revert: delete this file and run `sudo sysctl --system`.
vm.dirty_ratio = 40

sudo sysctl --system
sysctl vm.dirty_ratio

vm.dirty_ratio = 40

Four lines of comment. Date, reason, evidence, revert instruction. Without them, the next person finds an unexplained non-default value and faces a choice between leaving it forever and breaking something by removing it - so it stays forever, and the host accumulates settings nobody dares touch.

The sysctl block copied from a tuning guide

Every few years a “Linux performance tuning” post circulates with thirty sysctl lines and no measurements. Pasting it into /etc/sysctl.d/ is the most reliable way to make a host worse in ways that take a year to find.

Three specific reasons:

The defaults moved. Many values in circulating guides were written against kernels where the default was much lower. Some of those “increases” are decreases today. Always read the current value before writing a new one - if your new number is smaller than what is there, stop.
The workload was not yours. A setting that helps a 10 Gb bulk-transfer host hurts a latency-sensitive request/response host, and vice versa. There is no universally good value for a buffer size.
Nothing is attributable afterwards. Thirty changes made at once cannot be bisected. When something regresses, the only honest remediation is to remove all thirty and start over.

If you inherit such a file, do not delete it during an incident. Record the current values, then remove entries one at a time across change windows, measuring each.

The loop, written out

1. State the symptom in RED terms.
     "p99 checkout latency went from 180 ms to 900 ms on 9 Aug."
2. Measure. Collect a distribution, not a number.
     Baseline: n=30, p50=210, p95=880, max=1400
3. Apply USE to find the resource that could explain it.
     Disk aqu-sz 24 with await 40 ms, up from 0.6 ms. CPU idle.
4. Form one hypothesis that predicts something.
     "The mq-deadline scheduler is adding queueing on this NVMe.
      Switching to none should cut p95 by more than the run-to-run
      spread of 120 ms."
5. Change one variable. Runtime only.
     echo none | sudo tee /sys/block/nvme0n1/queue/scheduler
6. Re-measure the same way, with a control.
     After: n=30, p50=190, p95=610, max=980. Control host unchanged.
7. Keep or revert. If keeping, persist it WITH the evidence.
8. Write down what you learned, including the changes that did
   nothing. Those are the expensive ones to rediscover.

Step 8 is the one everybody skips and the one that compounds. A team that records its negative results stops re-testing the same useless knob every eighteen months.

Never tune during the incident, unless the tuning is the mitigation

During an outage the pressure to “try something” is enormous, and a change made then is the worst kind: unmeasured, unrecorded, and made simultaneously with three other people’s changes.

Separate mitigation from tuning. Mitigation is restoring service - failing over, restarting, shedding load, rolling back a deploy - and it is legitimate to do it fast and without a baseline. Tuning is improving a system that is currently working, and it belongs in a change window with a measurement on either side.

If you must change a tunable during an incident, use sysctl -w or a /sys write so it is runtime-only, and say so in the incident channel with the old value. That single message is what lets the review reconstruct what happened.