Benchmarking the workload you actually have

Advanced⏱ ~18 minfiorbdblktrace

What you'll learn

Characterise a production workload
Translate that characterisation into a benchmark
Recognise when a synthetic profile is inadequate
Validate the benchmark against production behaviour

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

“4k random write” is a benchmark profile, not a workload. Real workloads mix sizes, alternate between patterns, and have idle periods — and the differences change the answer.

Characterising a real workload

# from the RBD layer
rbd perf image iostat --pool rbd-vms

# per-image detail over time
rbd perf image iotop

# from the guest, the most accurate source
blktrace -d /dev/vdb -o - | blkparse -i - | \
  awk '$6=="Q" {print $7, $10}' | sort | uniq -c | sort -rn | head

# simpler, from iostat inside the guest
iostat -x 5 12 /dev/vdb

The properties to establish:

Property	Why it matters
Read/write ratio	writes cost replication; reads do not
Block size distribution	not a single value
Random versus sequential proportion	different device behaviour
Queue depth over time	determines the operating point
Idle periods	caches and background work behave differently
Working set size	determines cache effectiveness

Translating into a benchmark

[global]
ioengine=rbd
pool=rbd-vms
rbdname=bench
direct=1
time_based=1
runtime=600
ramp_time=60
percentile_list=50:95:99:99.9

[db-like]
rw=randrw
rwmixread=70
bssplit=4k/60:8k/25:16k/10:64k/5
iodepth=16
numjobs=4

bssplit reproduces a block size distribution rather than a single value, which is usually the largest single improvement in fidelity.

# with a duty cycle, closer to a real application
[bursty]
rw=randwrite
bs=4k
iodepth=32
thinktime=50ms
thinktime_blocks=100

When a synthetic profile is inadequate

Workload	Why synthetic fails
Database	correlated access patterns, fsync timing
Filesystem metadata	tiny operations, dependencies
Backup	large sequential with periodic sync
Mixed VM fleet	many independent streams

For these, replaying a captured trace or running the actual application against a test volume gives a far better answer than any synthetic approximation.

# capture and replay
blktrace -d /dev/vdb -o trace
fio --read_iolog=trace.log ...

Validating against production

# does the benchmark produce similar cluster-side characteristics?
ceph osd pool stats rbd-vms       # during production
ceph osd pool stats bench          # during the benchmark

If the operation rate, read/write mix, and average object size differ substantially between the two, the benchmark is not reproducing the workload and its results will not predict it.

Quiz

Knowledge check · 4 questions

Q1. Why do synthetic benchmarks poorly predict database performance?

Databases issue I/O through their own storage layer, and no fio engine reproduces the request stream that layer emits Databases bypass the page cache entirely with O_DIRECT, so every read reaches the cluster while a benchmark's reads do not Database throughput is bounded by log durability — small writes each followed by a flush and a wait, which a fixed-queue-depth benchmark never reproduces fio cannot generate writes smaller than 4k, while databases commit log records considerably smaller than that
Q2. Reproducing a workload's block size distribution usually improves a benchmark's predictive value more than any other single change.

True False
Q3. Build a representative benchmark.

A benchmark using 4k random write at qd32 predicted 8,000 IOPS for a new database deployment. In production the database achieves 2,000 transactions per second with much worse latency than predicted.
Q4. How do you validate that a benchmark reproduces a production workload?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Characterise the real workload’s block size distribution, read/write mix, and actual queue depth before writing a benchmark profile, and use bssplit rather than a single size. For flush-bounded workloads such as databases, run the application itself against a test volume — no fio profile reproduces the pattern.

Cross-course references

Kubernetes: load tests using uniform requests mispredict real traffic identically
Linux: application-level benchmarks beat synthetic ones whenever the pattern is correlated