Building a test cluster that predicts production

Why this matters in production

There is no upgrade rollback, so the test cluster is the only place a problem can be discovered cheaply — but only if it resembles production in the ways that matter.

What must match

# capture what production runs, to replicate
ceph versions
ceph mgr module ls --format json | python3 -c '
import sys,json; print(json.load(sys.stdin)["enabled_modules"])'
ceph osd pool ls detail | grep -E '^pool'
ceph orch ls
ceph features

What may differ

The caveat: an upgrade on an empty cluster does not exercise PG peering
and recovery on daemon restart. A small but non-zero data set is worth
having.

# put some data in the test cluster
rados bench -p testpool 60 write --no-cleanup

Running a representative test

# 1. bring the test cluster to production's version
TARGET=19.2.1
ceph version

# 2. replicate the production config that matters
ceph config dump   # compare against the production capture

# 3. apply representative load during the upgrade
fio --name=load --ioengine=rbd --pool=testpool --rbdname=testimg \
    --rw=randrw --bs=4k --iodepth=32 --numjobs=2 --time_based \
    --runtime=3600 --direct=1 &

# 4. run the upgrade
ceph orch upgrade start --ceph-version ${TARGET}

# 5. observe
watch -n 30 'ceph orch upgrade status; ceph -s | head -20'

Running load during the upgrade is what distinguishes a test from a
demonstration. An upgrade on an idle cluster proves the daemons restart;
an upgrade under load proves clients survive it.

What the test proves and does not

Duration is the attribute that scales: a three-host test says nothing
about how long thirty hosts take. Estimate from the daemon count.

Quiz

Production discipline

Match the test cluster’s configuration — Ceph version, deployment method, enabled mgr modules, pool types, service types — rather than its hardware, and run client load throughout the test upgrade. Estimate production duration from daemon count; the test’s wall clock does not scale.

Cross-course references

• Kubernetes: staging clusters predict upgrades only when the addon set matches
• Linux: a test that does not exercise the workload proves only that services start