Establishing a performance baseline

Why this matters in production

Without a baseline, every performance report is an opinion about whether the current number is bad. With one, it is arithmetic.

Designing a reproducible baseline

# cluster-level, from the RADOS layer
rados bench -p bench 60 write --no-cleanup
rados bench -p bench 60 seq
rados bench -p bench 60 rand
rados -p bench cleanup

# client-level, the numbers that match user experience
fio --name=baseline-randwrite --ioengine=rbd --pool=rbd-vms \
    --rbdname=bench --rw=randwrite --bs=4k --iodepth=32 \
    --runtime=120 --time_based --percentile_list=50:95:99:99.9

fio --name=baseline-seqread --ioengine=rbd --pool=rbd-vms \
    --rbdname=bench --rw=read --bs=4M --iodepth=16 \
    --runtime=120 --time_based

Four profiles cover most questions: 4k random write, 4k random read, 4M sequential write, 4M sequential read. More profiles are better; fewer than these leaves gaps.

Capturing the conditions

A number without its conditions is not reproducible:

Baseline: prod-ceph-01
Date:            2026-08-18
Ceph version:    20.2.0
Cluster state:   HEALTH_OK, all PGs active+clean
Cluster load:    idle (out of hours), 400 IOPS background
Hardware:        12× 16TB HDD/host, NVMe DB, 25GbE cluster net
Pool:            rbd-vms, size=3, 1024 PGs
Scheduler:       mclock, balanced profile
Client:          ceph-client-01, 32 cores, 25GbE

4k randwrite qd32:  8420 IOPS, p50 3.1ms, p95 7.4ms, p99 14.2ms
4k randread  qd32: 31200 IOPS, p50 0.9ms, p95 2.2ms, p99  5.1ms
4M seqwrite  qd16:  2.1 GiB/s
4M seqread   qd16:  3.8 GiB/s

The cluster state line matters most: a baseline taken during a recovery is not a baseline.

Storing baselines findably

# alongside the runbooks, in version control
git add docs/baselines/prod-ceph-01-2026-08-18.md

A baseline in someone’s home directory is a baseline that will not be found during an incident. The test is whether a colleague can locate it without asking.

Refreshing without losing history

baselines/
  prod-ceph-01-2026-02-01.md   # initial, pre-production
  prod-ceph-01-2026-05-14.md   # after the Squid upgrade
  prod-ceph-01-2026-08-18.md   # after adding 3 hosts

Keeping the old ones is what makes a gradual regression visible. A single overwritten baseline answers “is this normal now”; a series answers “when did this change”.

Quiz

Production discipline

Record the cluster state, version, hardware, and pool settings alongside every baseline number, and commit the file next to the runbooks so it is findable during an incident. Keep dated baselines rather than overwriting — the series is what makes a gradual regression visible.

Cross-course references

• Kubernetes: benchmark results without cluster conditions are equally unreusable
• Linux: any performance record needs its environment captured to stay valid