CephLXXII · BenchmarkingBenchmarking
fio against RBD
What you'll learn
- Write a fio job for RBD
- Configure percentile and distribution reporting
- Produce reproducible, comparable results
- Interpret the output correctly
Prerequisites
None — start here.
Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18
Why this matters in production
fio produces the numbers that support a decision: percentiles, distributions, and structured output that can be compared across runs and months.
A job file
[global]
ioengine=rbd
clientname=admin
pool=rbd-vms
rbdname=bench
direct=1
time_based=1
runtime=180
ramp_time=30
group_reporting=1
percentile_list=50:95:99:99.9
log_avg_msec=1000
[randwrite-4k-qd32]
rw=randwrite
bs=4k
iodepth=32
numjobs=1
[randread-4k-qd32]
rw=randread
bs=4k
iodepth=32
numjobs=1
stonewall
[seqwrite-4m-qd16]
rw=write
bs=4M
iodepth=16
numjobs=1
stonewallstonewall makes the jobs run sequentially rather than concurrently,
which is almost always what is wanted — concurrent jobs contend and the
results describe the contention rather than each workload.
fio rbd-bench.fio --output-format=json --output=results.jsonReading the output
python3 -c '
import json
d = json.load(open("results.json"))
for j in d["jobs"]:
for op in ("read","write"):
s = j[op]
if not s["io_bytes"]: continue
p = s["clat_ns"]["percentile"]
print("%-24s %-6s %7d IOPS p50 %6.2f p99 %7.2f p99.9 %8.2f ms" % (
j["jobname"], op, s["iops"],
p["50.000000"]/1e6, p["99.000000"]/1e6, p["99.900000"]/1e6))'| Field | Meaning |
|---|---|
clat | completion latency — the useful one |
slat | submission latency — usually negligible |
lat | total, slat + clat |
iops | operations per second |
bw | bandwidth |
Percentiles come from clat, and quoting lat where clat was meant is
a common minor error.
Reproducibility
# record everything that could affect the result
{
echo "date: $(date -Is)"
echo "ceph: $(ceph version)"
echo "health: $(ceph health)"
echo "fio: $(fio --version)"
echo "client: $(hostname), $(nproc) cores"
ceph osd pool get rbd-vms size
ceph config get osd osd_mclock_profile
} > run-metadata.txtA result without this metadata cannot be compared with a result from six months later, because any of these could have changed.
Interpreting correctly
| Observation | Reading |
|---|---|
| p50 low, p99 high | tail latency — an outlier or contention |
| p50 and p99 both high | uniformly slow — device or saturation |
| High variance between runs | interference; check for scrub or recovery |
| IOPS flat as iodepth rises | saturated |
| p99.9 far above p99 | rare severe events, often a scrub or a slow OSD |
Quiz
Knowledge check · 4 questions
Q1. What does `stonewall` do in a fio job file, and why does it matter?
Q2. Percentile figures should be taken from the `lat` field rather than `clat`.
Q3. Set up decision-grade benchmarking.
A team needs benchmark results that can be compared against runs six months from now, after a planned upgrade and hardware refresh.
Q4. Why does `direct=1` matter when benchmarking with the rbd ioengine?
Passing score: 75%. Answers are checked in this browser.
Production discipline
Commit the fio job file to version control and capture Ceph version,
health, pool settings, and client hardware with every run — comparability
across months depends on both. Use stonewall, direct=1, time_based,
and ramp_time; each removes a specific way the numbers can mislead.
Cross-course references
- Kubernetes: benchmark manifests in version control give the same reproducibility
- Linux: any performance comparison needs the environment captured alongside the numbers