Throughput and saturation

Why this matters in production

Saturation is where throughput stops rising and latency starts. Knowing which resource saturated determines what to do about it.

Recognising saturation

for qd in 1 4 16 64 256; do
  fio --name=q$qd --ioengine=rbd --pool=rbd-vms --rbdname=bench \
      --rw=read --bs=4M --iodepth=$qd --runtime=30 --time_based \
      --output-format=json | python3 -c '
import sys,json; d=json.load(sys.stdin)["jobs"][0]["read"]
print("qd %-4s %8.1f MiB/s  %7.2f ms" % ("'"$qd"'", d["bw"]/1024, d["lat_ns"]["mean"]/1e6))'
done

qd 1      412.0 MiB/s     9.71 ms
qd 4     1580.0 MiB/s    10.12 ms
qd 16    2280.0 MiB/s    28.05 ms
qd 64    2310.0 MiB/s   110.60 ms
qd 256   2315.0 MiB/s   441.90 ms

Between qd16 and qd64 throughput stops rising and latency rises in proportion. That is saturation, and qd16 is the useful operating point.

Identifying the saturated resource

# devices
IFACE=bond0
iostat -x 1 5      # on OSD hosts: %util, aqu-sz

# network
sar -n DEV 1 5     # rxkB/s, txkB/s against link capacity
ethtool ${IFACE} | grep Speed

# CPU
mpstat -P ALL 1 5  # on both OSD hosts and the client

# OSD-internal
ceph osd perf | sort -k2 -rn | head -5

The last row is important: flat throughput with nothing saturated means the client is not asking for more.

Distinguishing saturation from other limits

# is it one OSD or all of them?
ceph osd perf | awk 'NR>1 {print $2}' | sort -n | \
  awk '{a[NR]=$1} END {print "median", a[int(NR/2)], "max", a[NR]}'

Reporting honestly

Naming the saturated resource is what makes the next step obvious rather than a guess.

Quiz

Production discipline

Name the saturated resource alongside any throughput figure — device, network, CPU, or client concurrency — because it determines whether more hardware helps and which hardware. Flat throughput with nothing saturated means the benchmark itself is the limit.

Cross-course references

• Kubernetes: identifying the saturated resource decides whether to scale out or up
• Linux: the USE method names utilisation, saturation, and errors for exactly this