Benchmarking without disturbing production

Intermediate⏱ ~17 mincephfio

What you'll learn

Isolate a benchmark from production workloads
Recognise when a benchmark is invalidated by interference
Choose an appropriate environment
Bound the impact of an unavoidable production benchmark

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

A benchmark on a production cluster affects production, and production affects the benchmark. Both directions matter and only one is usually considered.

Levels of isolation

Approach	Isolation	Fidelity to production
A dedicated test cluster	complete	depends on how well it matches
A dedicated CRUSH root on the same hardware	high	high
A dedicated pool sharing OSDs	none	complete
A production pool	none	complete

The middle option is often the practical compromise: separate OSDs, same hardware, same network.

ceph osd crush add-bucket bench-root root
ceph osd crush move ceph-test-01 root=bench-root
ceph osd crush rule create-replicated bench-rule bench-root host
ceph osd pool create bench 128 128 replicated bench-rule

Recognising interference

# before the benchmark
ceph -s
ceph health detail
ceph osd dump | grep flags
ceph pg dump pgs | grep -c scrubbing

Condition	Effect on the benchmark
Recovery running	results reflect contention
Deep scrub active	results reflect contention
Another benchmark running	results reflect both
Production load varying	results vary between runs

# quiesce what can be quiesced
ceph osd set noscrub
ceph osd set nodeep-scrub
# wait for in-progress scrubs to drain
watch -n 30 'ceph pg dump pgs | grep -c scrubbing'

Restoring the flags afterwards is essential and easy to forget:

ceph osd unset noscrub
ceph osd unset nodeep-scrub

Bounding the impact when production cannot be avoided

# limit the benchmark's own concurrency
fio --iodepth=4 --numjobs=1 ...

# and its duration
fio --runtime=60 --time_based ...

# watch production latency during it
watch -n 5 'ceph osd pool stats rbd-vms'

# an abort condition, agreed in advance
# if production p99 exceeds X, stop the benchmark

Agreeing the abort condition before starting is what makes a production benchmark defensible.

Reporting honestly

Benchmark: prod-ceph-01, 2026-08-18 02:00
Isolation: dedicated pool, shared OSDs with production
Production load during run: 3,200 IOPS average (typical overnight)
Scrubs: disabled for the duration
Recovery: none
Caveat: results reflect a cluster carrying overnight production load;
        peak-hour capability will be lower

The caveat is what prevents the number being quoted later as the cluster’s capability.

Quiz

Knowledge check · 4 questions

Q1. Why does a dedicated pool on shared OSDs provide no performance isolation?

Its PGs map onto the same OSDs, so benchmark and production I/O queue behind each other at every device Pools cannot be given separate scheduler settings, so both inherit the same mClock profile The monitors serialise access between pools, so only one pool's operations are dispatched at a time CRUSH ignores pool boundaries when placing PGs, so a new pool lands wherever there is free space
Q2. Disabling scrub for a benchmark is reasonable, and the risk lies almost entirely in what happens afterwards.

True False
Q3. Benchmark a production cluster.

A capability measurement is needed on a production cluster. No test cluster exists and no OSDs can be dedicated. The benchmark must run against production hardware carrying live load.
Q4. Why must a production benchmark result be reported with the concurrent production load?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Script the restoration of noscrub and nodeep-scrub alongside the benchmark rather than relying on memory; the flags normalise as expected and PGs then go unscrubbed for weeks. Record the concurrent production load with any result obtained on a live cluster, so it is not later quoted as the cluster’s capability.

Cross-course references

Kubernetes: load testing against production needs the same abort conditions
Linux: benchmarking a live system always requires stating what else was running