Post-maintenance verification

Why this matters in production

The work finishing and the cluster being restored are different things, and the gap between them is where the next incident begins.

Cluster state

ceph -s
ceph health detail

Expected:
  HEALTH_OK
  all OSDs up and in, at the expected count
  all PGs active+clean
  no flags set
  monitor quorum complete
  managers as expected

ceph osd stat
ceph pg stat
ceph osd dump | grep flags
ceph quorum_status --format json | python3 -c '
import sys,json; print(json.load(sys.stdin)["quorum_names"])'
ceph mgr stat
ceph orch ps --refresh | grep -v running

The last command lists any daemon not running, which catches a service that did not restart cleanly.

Performance against the baseline

# per-OSD, on the maintained host
for osd in $(ceph osd ls-tree ceph-03); do
  printf 'osd.%-4s ' "$osd"
  ceph tell osd.$osd bench 12288000 4096 4194304 100 2>/dev/null | \
    python3 -c 'import sys,json; print(round(json.load(sys.stdin)["iops"],1))'
done

# client-observed, against the recorded baseline
# Substitute your own pool before running:
POOL=rbd-vms

fio --name=post --ioengine=rbd --pool="$POOL" --rbdname=probe \
    --rw=randwrite --bs=4k --iodepth=32 --runtime=180 --time_based \
    --percentile_list=50:95:99

Comparing against the pre-maintenance figures is what distinguishes a clean return from a silent regression.

Residue

# the config capture taken before the maintenance window
CONFIG_BEFORE=/tmp/config-before-2026-08-18.txt

diff <(ceph config dump) "$CONFIG_BEFORE"

Diffing against the captured before state surfaces every configuration difference at once, including ones nobody remembers making.

Closing the record

Maintenance: ceph-03 kernel upgrade
Planned:     22:00–22:30
Actual:      22:00–22:41
Result:      completed
Verification:
  HEALTH_OK at 22:44
  96 OSDs up and in
  all PGs active+clean
  no flags set
  osd bench within 3% of pre-maintenance figures
  client p99 4.1 ms against a 4.0 ms baseline
Residue:     none; config diff clean
Notes:       overran by 11 minutes due to a slow POST

The overrun note is what improves the next window’s estimate.

Quiz

Production discipline

Diff ceph config dump against the state captured before the maintenance and run ceph orch ps for any daemon not running — HEALTH_OK coexists with left-adjusted throttles, a stopped RGW, and a performance regression. Record the actual duration; it is what improves the next estimate.

Cross-course references

• Kubernetes: verifying every component restarted after a node operation, not just readiness
• Linux: post-change verification against a captured baseline is what closes a change