Recovery paths that do not depend on what failed

Advanced⏱ ~18 mincephtraceroute

What you'll learn

Trace the network path between cluster and backup target
Identify circular dependencies in the recovery path
Hold the configuration a rebuild needs outside the cluster
Place the tiebreaker path independently of the inter-site link

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

The question is not whether the backup exists. It is whether you can reach it, authenticate to it, and find out what to do with it while the cluster is off.

Tracing the path

ceph config get mon public_network
ceph config dump | grep -E 'public_network|cluster_network'

ip route get 10.90.0.10
traceroute -n 10.90.0.10

A different subnet is not a different path. Compare the hop lists: if the cluster’s public network and the backup path traverse the same core switches, firewall pair or upstream router, one control-plane failure takes both.

Shared component	Consequence when it fails
top-of-rack pair	the cluster and its local backup target vanish together
core switches	the offsite copy becomes unreachable during the event
firewall pair	the restore path is blocked by the outage it exists for
DNS resolvers	endpoints stop resolving; addresses have to be known
the inter-site link	in a stretch design, the thing the tiebreaker adjudicates

Circular dependencies

Recovery-path dependency	Fails with the cluster when
DNS	resolvers run on VMs backed by this cluster
the credential vault	its storage is this cluster’s RGW
the backup catalogue	it lives on this cluster’s CephFS
host provisioning and images	artefacts are in this cluster’s buckets
monitoring you would verify with	its metrics are stored here
the runbook	the wiki is backed by this cluster

The test is one sentence: can the restore be completed with the cluster
powered off? Walk the procedure and mark every step that would fail.

What to hold outside the cluster

ceph orch ls --export > /secure/offsite/service-specs.yaml
ceph config dump > /secure/offsite/config-dump.txt
ceph osd getcrushmap -o /secure/offsite/crush.bin
ceph auth export -o /secure/offsite/auth.export

ceph osd pool ls detail > /secure/offsite/pools.txt
ceph fs dump > /secure/offsite/fs-dump.txt

The auth export contains every key in the cluster. It belongs under the
same controls as any credential, and it belongs somewhere the cluster
cannot reach.

Verifying from outside

# from a host on the backup path, with the cluster assumed gone
dig +short backup.example.net
curl -sf -o /dev/null -w '%{http_code}\n' https://backup.example.net/healthz

Quiz

Knowledge check · 4 questions

Q1. What is the single test that exposes circular dependencies in a recovery path?

Whether the restore can be completed with the cluster powered off Whether the backup target sits on a subnet the cluster does not route to Whether last night's backup completed and its checksum was verified Whether the backup target has capacity for a full estate restore
Q2. A backup target reachable over a different IP subnet is network-disjoint from the cluster.

True False
Q3. Audit a recovery path for shared dependencies.

Backups land on an object target in a second building. Its endpoint resolves through the site DNS servers, which run as VMs on RBD in the cluster being protected. The backup credentials live in a vault whose storage backend is this cluster's RGW.
Q4. What cluster configuration should be held outside the cluster for a rebuild?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Walk the restore procedure with the cluster assumed powered off and mark every step that consults something the cluster hosts — DNS, the vault, the catalogue, the runbook. Export the service specs, config, CRUSH map and auth to offline storage; a rebuild that begins by rediscovering the design has already missed its RTO.

Cross-course references

Kubernetes: a disaster runbook stored in a cluster-hosted wiki is unavailable during the disaster
Linux: independence has to be verified on the physical path, not inferred from addressing