Benchmarks that mislead

Advanced⏱ ~18 minfiorados

What you'll learn

Recognise the common benchmarking errors
Explain why each produces a misleading number
Design a benchmark that avoids them
Challenge a benchmark result constructively

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Most benchmarks that disagree with production do so for one of a small number of reasons, and recognising them is faster than re-running.

The common errors

Error	Why it misleads
Working set fits in cache	measures RAM, not storage
Run too short	measures burst, not sustained
No ramp-up exclusion	includes cold-cache warm-up
Single queue depth reported	one point on a curve
Averages only	conceals the tail
Benchmark on an idle cluster	production is not idle
Writes to a thin image never written	measures allocation, not overwrite
Reads of data just written	served from cache at every layer
Compressible or zero data	compression and dedup distort it

The cache error

# a 10 GiB image with 128 GiB of client RAM
fio --ioengine=rbd --rbdname=small-image --size=10G --rw=randread ...

The entire image fits in cache after the first pass, so subsequent reads measure memory. The fix is a working set several times larger than every cache in the path, plus direct=1:

rbd create --size 2T rbd-vms/bench
fio ... --direct=1 --size=2T

The thin-provisioning error

rbd create --size 1T rbd-vms/bench
fio --rw=randwrite ...     # first write to each object

The first write to an RBD object allocates it, which is a different and often faster path than overwriting existing data. Pre-filling the image before measuring is what makes the result represent steady state:

fio --name=prefill --ioengine=rbd --pool=rbd-vms --rbdname=bench \
    --rw=write --bs=4M --iodepth=16 --size=1T
# then measure

The compressible data error

# fio writes zeros by default in some configurations
fio ... --refill_buffers --buffer_compress_percentage=0

Zero-filled or highly compressible data is handled differently by compression, and the result does not represent real data. Explicit random buffers avoid this.

The idle-cluster error

A benchmark on an idle cluster measures the cluster with nothing else happening, which no production system ever is. The result is an upper bound rather than a prediction, and should be reported as one.

Challenging a result constructively

Questions worth asking of any benchmark number:
  What block size and access pattern?
  What queue depth, and was a curve measured?
  What was the working set relative to cache?
  Was the image pre-filled?
  What percentiles, not just the mean?
  What else was running on the cluster?
  How long did it run, and was ramp-up excluded?

Quiz

Knowledge check · 4 questions

Q1. Why must a benchmark image be pre-filled before measuring random writes?

To confirm every object in the image is allocated and readable, so a read error cannot interrupt the run The first write to an object allocates it, which is a different and often faster path than overwriting existing data To warm the client and OSD caches, so the measured run reflects steady-state cache hit rates To settle the CRUSH mapping, since placement for an image is fixed only once its objects exist
Q2. Setting `direct=1` on the client is sufficient to ensure reads reach the storage media.

True False
Q3. Challenge a benchmark result.

A vendor demonstration shows 400,000 IOPS random read on a small cluster. The demonstration used a 20 GiB image on hosts with 256 GiB RAM each, running for 30 seconds.
Q4. List four questions worth asking of any benchmark result.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Pre-fill benchmark images and use a working set several times the aggregate cache in the path; without both, the result measures allocation and memory rather than storage. Ask the standard set of questions of any benchmark number before acting on it — most misleading results fail one of them.

Cross-course references

Kubernetes: load tests with unrealistic data or duration mislead identically
Linux: benchmarking a filesystem without exceeding page cache is the classic version