What to capture from a benchmark

Intermediate⏱ ~17 minfioceph

What you'll learn

Enumerate what a benchmark record must contain
Capture latency distribution rather than averages
Store results so they are found and comparable
Present results without misleading

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

A benchmark result is used months later by someone who was not there. What they need is rarely what was recorded.

The record

Benchmark record
================
Date:            2026-08-18T02:14:00Z
Operator:        <name>
Purpose:         baseline before Tentacle upgrade

Cluster
  Name:          prod-ceph-01
  Ceph version:  20.2.0
  Health:        HEALTH_OK, all PGs active+clean
  OSDs:          96 HDD + 12 NVMe
  Scheduler:     mclock, balanced
  Concurrent load: 3,200 IOPS (overnight typical)

Pool
  Name:          bench
  Type:          replicated, size=3, min_size=2
  PGs:           128
  CRUSH rule:    hdd-rule

Client
  Host:          ceph-client-01
  Cores / RAM:   32 / 128 GiB
  Network:       25 GbE
  fio version:   3.35

Results
  Profile              IOPS    p50      p95      p99      p99.9    BW
  4k randwrite qd32    8420    3.1 ms   7.4 ms   14.2 ms  48.1 ms  33 MiB/s
  4k randread  qd32   31200    0.9 ms   2.2 ms    5.1 ms  19.4 ms 122 MiB/s
  4M seqwrite  qd16     540    —        —        —        —       2.1 GiB/s
  4M seqread   qd16     970    —        —        —        —       3.8 GiB/s

Caveats
  Scrub disabled for the duration; restored afterwards.
  Overnight production load present; peak-hour figures will be lower.

Why percentiles rather than averages

configuration A: mean 4.1 ms, p99  9 ms
configuration B: mean 4.0 ms, p99 87 ms

Identical averages, entirely different user experience. Recording only the mean makes the two indistinguishable in the record, and the choice between them unrecoverable later.

fio ... --percentile_list=50:95:99:99.9 --output-format=json

Storing it

git add docs/baselines/prod-ceph-01-2026-08-18.md
git add docs/baselines/prod-ceph-01-2026-08-18.json
git commit -m "baseline: prod-ceph-01 before Tentacle upgrade"

Both forms: the readable record for humans and the JSON for tooling. Kept next to the runbooks, both are found by someone who does not know they exist.

Presenting without misleading

Statement	Problem
“The cluster does 31,000 IOPS”	which profile, which queue depth, which pool
“Latency is 3 ms”	which percentile
“It got faster after the upgrade”	compared against what, under what load
“31,200 IOPS 4k random read at qd32, p99 5.1 ms, on the HDD pool”	complete

The complete form is longer and is the only one that can be checked.

Quiz

Knowledge check · 4 questions

Q1. What does a large gap between p99 and p99.9 indicate?

A measurement error, since a percentile beyond p99 needs more samples than a short run collects Rare but severe events — a scrub starting, an OSD stalling — that are invisible in the p99 Too short a run, so the last few operations were still ramping up when the run ended Client-side queueing, since the deepest queue occupancy occurs only at the very end of a run
Q2. Recording the mean latency is sufficient for comparing two configurations.

True False
Q3. Review an existing benchmark record.

A benchmark record from last year states "cluster achieves 31,000 IOPS with 3 ms latency". The team wants to compare current performance against it after a hardware refresh.
Q4. What makes a performance statement complete rather than misleading?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Capture p50, p95, p99, and p99.9 as a minimum; means conceal the tail and two configurations with identical means can differ tenfold where users notice. Commit both a readable record and the JSON next to the runbooks, along with the job file, so a later comparison is possible rather than suggestive.

Cross-course references

Kubernetes: SLO records need percentiles for exactly the same reason
Linux: any performance claim without its parameters cannot be verified