Rebuilding a cluster from nothing

Why this matters in production

A cluster rebuild is the recovery of last resort, and it is the one whose duration is most consistently underestimated.

The sequence

1. Provision hosts and network
2. Bootstrap a new cluster
3. Add hosts and deploy daemons
4. Recreate the CRUSH topology
5. Recreate pools with their parameters
6. Recreate cephx entities from the auth backup
7. Restore data from backup
8. Recreate service layers — CephFS volumes, RGW zones
9. Repoint clients
10. Verify

# 2
IP=10.20.0.11
CIDR=10.30.0.0/24
cephadm bootstrap --mon-ip ${IP} --cluster-network ${CIDR}

# 3
HOST=stor-04
IP=10.20.0.11
ceph orch host add ${HOST} ${IP} --labels _admin
ceph orch apply osd --all-available-devices
ceph orch host ls

Configuration before data

# 4 — the topology, from the backup
crushtool -c /backups/ceph-pre/crush.txt -o /tmp/crush.bin
ceph osd setcrushmap -i /tmp/crush.bin
ceph osd tree

# 5 — pools, from the recorded detail
ceph osd pool create tenant-acme 128
ceph osd pool application enable tenant-acme rbd
ceph osd pool set tenant-acme size 3
ceph osd pool set-quota tenant-acme max_bytes 10995116277760

# 6 — entities, from the auth export
ceph auth import -i /backups/ceph-pre/auth-export.txt
ceph auth ls | head

# 8 — service layers
ceph fs volume create cephfs
ceph orch apply rgw acme --realm=acme --zone=site-a

Restoring configuration is minutes. Everything after it is bounded by
data volume.

Estimating realistically

Rebuild duration =
    host provisioning
  + cluster bootstrap and daemon deployment
  + configuration restore                    (minutes)
  + data restore                             (dominant)
  + client repointing and verification

# the data volume that must move
ceph df   # on the old cluster, if reachable, or from the backup inventory

The honest figure for a multi-hundred-terabyte cluster is days, and
stating it before an incident is far easier than during one.

Verifying

ceph -s
ceph osd tree
diff <(ceph osd pool ls detail) /backups/ceph-pre/pools.txt
diff <(ceph auth ls --format json | python3 -c '
import sys,json
for e in sorted(json.load(sys.stdin)["auth_dump"], key=lambda x: x["entity"]):
    print(e["entity"])') /backups/ceph-pre/entities.txt

# and the data
POOL=rbd-vms
rbd ls ${POOL} | wc -l
ceph df detail

# clients actually work
ceph -n client.app-a --keyring /etc/ceph/ceph.client.app-a.keyring -s

Quiz

Production discipline

State the realistic rebuild figure before an incident — configuration restore is minutes and data restore is days for a large cluster. Plan for client repointing as real work; every reference to the old fsid, monitor addresses, and cluster ID must be updated.

Cross-course references

• Kubernetes: a rebuilt cluster needs every kubeconfig and CA reference updated
• Linux: rebuild time is dominated by data, and reconnection is discovered rather than planned