Testing a workload against Ceph before committing to it

Intermediate⏱ ~17 minfioradosceph

What you'll learn

State the four properties that predict a good fit
Characterise an existing workload before migrating it
Reproduce the workload profile against a test pool
Justify consolidation onto one hardware pool

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Ceph fits workloads with a particular shape, and the shape is measurable before anything is migrated.

The four properties

Property	Threshold that matters	Why
Concurrency	many clients or deep queues	aggregate scales; a single operation does not
Latency tolerance	roughly 0.5 ms and above for writes	the write path crosses the network twice
Capacity trajectory	growing over the asset life	the fixed cost amortises over growth
Failure domain requirement	a real one, written down	CRUSH is the reason to be here

Three of the four are about the workload. The fourth is about whether
anyone has actually decided what the storage must survive.

Characterising the existing workload

# on the current storage, at a representative peak
iostat -x 5 12

# the numbers that decide fit
#   avgqu-sz / aqu-sz : concurrency the workload presents
#   r_await / w_await : the latency it currently tolerates
#   rareq-sz / wareq-sz : request size, which decides IOPS or bandwidth

Observation	Implication
Queue depth consistently above 8	concurrency is available; a good sign
Queue depth 1 with low latency	the workload will feel every added microsecond
Write latency already 2 ms and accepted	headroom exists
Request size 4 KiB dominant	an IOPS problem
Request size 1 MiB dominant	a bandwidth problem, far easier to satisfy

Reproducing it against a test pool

rbd create --size 100G test/fit-probe
rbd map test/fit-probe

fio --name=fit --filename=/dev/rbd0 --direct=1 --ioengine=libaio \
  --rw=randwrite --bs=4k --iodepth=16 --numjobs=4 \
  --runtime=120 --time_based --group_reporting

# and the pessimistic case the aggregate figure conceals
fio --name=serial --filename=/dev/rbd0 --direct=1 --ioengine=libaio \
  --rw=randwrite --bs=4k --iodepth=1 --numjobs=1 \
  --runtime=120 --time_based

rados bench -p test 60 write -t 64 -b 4194304
rados -p test cleanup

Consolidation as a reason on its own

ceph osd crush class ls
ceph osd crush rule create-replicated fast default host nvme
ceph osd crush rule create-replicated bulk default host hdd

ceph osd pool create vms 256 256 replicated fast
ceph osd pool create archive 256 256 replicated bulk
ceph osd pool ls detail | grep -E 'pool|crush_rule'

Consolidated onto one cluster	Was
RBD for virtualisation	a SAN
CephFS for shared application state	a NAS
RGW for artefacts and backups	an object appliance
All three with per-class placement	three procurement cycles

Quiz

Knowledge check · 4 questions

Q1. Which observation from the existing storage most strongly predicts a good Ceph fit?

Total capacity above 100 TiB, which is the point at which the fixed cost of a cluster amortises Queue depth consistently above eight at peak, showing the workload presents concurrency A low current write latency, which shows the workload has headroom for the extra network traversal A large number of distinct filesystems, each of which becomes one CephFS volume after migration
Q2. Consolidating block, file, and object storage onto one hardware pool is a legitimate reason to choose Ceph even when no single interface justifies it on its own.

True False
Q3. Decide whether to migrate a workload to Ceph.

A team proposes moving a 40 TiB virtualisation estate off a SAN. The SAN reports 12000 write IOPS at peak with 0.9 ms average write latency.
Q4. Why does a workload running at queue depth one gain little from a larger Ceph cluster?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Measure per-client queue depth and request size before agreeing a migration — an aggregate IOPS figure from the current storage predicts nothing about the guests that will complain. Benchmark the pessimistic serialised case explicitly and compare it against what the sensitive workloads tolerate today.

Cross-course references

Kubernetes: horizontal scaling helps throughput and never helps a single slow request
Linux: queue depth and request size describe a workload better than any IOPS total