The design as a repository, and the acceptance test

Why this matters in production

A cluster nobody can rebuild from a repository is a cluster whose design exists only in the memory of whoever last touched it. The acceptance test is the last moment the design can be wrong for free.

What lives in Git

ceph orch ls --export                > specs/cluster.yaml
ceph config dump                     > config/ceph-config.dump
ceph osd pool ls detail              > pools/pools.txt
ceph osd getcrushmap -o /tmp/cm && crushtool -d /tmp/cm -o crush/crushmap.txt

# ceph auth ls prints secrets; strip them before this goes anywhere near Git
ceph auth ls | grep -v 'key:'        > auth/entities.txt

Specs are the deployment

# specs/osd-hdd.yaml
service_type: osd
service_id: hdd-with-nvme-db
placement:
  label: osd
spec:
  data_devices:
    rotational: 1
  db_devices:
    rotational: 0
  db_slots: 6

ceph orch apply -i specs/osd-hdd.yaml --dry-run
ceph orch apply -i specs/osd-hdd.yaml
ceph orch device ls --wide

Always --dry-run first. It prints the devices the spec would consume, which is the only way to find out that rotational: 1 also matched the drive you were about to send back to the vendor.

Drift detection

diff <(ceph orch ls --export) specs/cluster.yaml
diff <(ceph config dump)      config/ceph-config.dump
diff <(ceph osd pool ls detail) pools/pools.txt

Run it nightly and alert on a non-empty diff. Drift is not always wrong — it is always unrecorded, and the two become the same thing during a rebuild.

The recovery kit

ceph mon getmap -o kit/monmap.bin
ceph auth get client.admin -o kit/ceph.client.admin.keyring
ceph osd getcrushmap -o kit/crushmap.bin
ceph config dump > kit/config.dump

The kit holds secrets, so it lives outside Git, encrypted, in a location with different credentials and a different failure domain from the cluster. A recovery kit stored on CephFS is not a recovery kit.

The acceptance test

# 1 layout matches the design
ceph orch ls && ceph -s && ceph osd tree

# 2 no slow drives hiding in the batch
for i in $(ceph osd ls); do
  printf '%-6s ' "osd.$i"
  ceph tell osd.$i bench 12288000 4096 -f json | python3 -c '
import sys,json; print("%.1f MB/s" % (json.load(sys.stdin)["bytes_per_sec"]/1e6))'
done

# 3 throughput against a throwaway pool
ceph osd pool create bench 128 128 replicated on-hdd
rados bench -p bench 300 write -b 4M -t 64 --no-cleanup
rados bench -p bench 120 rand -t 64
ceph osd pool delete bench bench --yes-i-really-really-mean-it

# 4 the failure drill: power off one host and watch
ceph -s                          # degraded, no inactive PGs
ceph config get osd osd_pool_default_min_size
watch -n 30 ceph -s              # recovery starts after mon_osd_down_out_interval

A design that has not lost a host on purpose has not been tested. Do it while the cluster is empty and the only cost is an afternoon.

Quiz

Production discipline

Export the cluster into Git nightly and alert on any diff — drift is not necessarily wrong, but it is unrecorded, and during a rebuild those are the same thing. Run the host-loss drill before the first workload lands; it is the only test that exercises the failure domain, and it is free exactly once.

Cross-course references

• Kubernetes: GitOps drift detection is the same nightly diff against a declared state
• Linux: a backup of the credentials needed to restore, stored on the system being restored, is not a backup