Recovering a cluster onto clean infrastructure

The previous lesson drew a boundary around what a workload-level backup tool covers and named the places its guarantees stop. This lesson takes the scenario those limits are ultimately measured against: the cluster is gone. Not a namespace, not a node — the control plane, the workers, the etcd data directory and every address that pointed at them. A new cluster has to carry the workload by tonight, and the order in which it is assembled decides whether the result is a recovery or a convincing imitation of one.

The steps below are not a matter of taste. Each one is where it is because performing it earlier makes the next step wrong in a way that nothing reports.

Five steps, and why the order is not negotiable

The first step is to build the cluster itself from its own definition — the installer configuration, the node roles and counts, the container network plugin, the storage classes, the ingress controller. That is infrastructure work, and it produces an empty but complete platform.

The second is to put the desired state back, either by re-applying the declared manifests or by restoring a cluster-state snapshot. This is the step most people picture when they say “restore the cluster”, and it is the second of five.

The third is to restore the persistent data into the volumes the re-created claims have just been given, because the data is a separate artefact with a separate lifecycle. The fourth is to reconcile the dependencies that live outside the cluster: the image registry and the credentials used to pull from it, the secrets held in an external store, the certificate issuer, the DNS records and ingress addresses the outside world resolves, and the cloud resources the manifests name. The fifth is to validate a business transaction, not a set of conditions.

Now consider what happens when the order is disturbed. Apply the manifests before the storage classes exist and the claims have nothing to bind to. Restore the data before the claims exist and the bytes land in a directory that no volume will ever be backed by. Reconcile the external dependencies after declaring victory and the first real request is what discovers the registry is unreachable. Validate on readiness and you learn that a container started, which — as the capture below shows — is compatible with the service being completely wrong.

The cluster is rebuilt from a definition, not from a snapshot

The temptation at hour zero is to stand up any cluster that will accept a kubectl connection and pour the state into it. That inverts the dependency. The state describes objects that refer to a specific cluster shape: node names, storage class names, ingress class names, load balancer addresses, the signing key behind every ServiceAccount token. Restoring into a differently shaped cluster reinstates references to things that are not there.

The identity mechanics were measured directly in the etcd capture. Restoring a snapshot with an --initial-cluster identical to the original reproduced the cluster id 1c45a069f3a1d796; restoring the same file with a different membership produced 80d54574493dd420. The identity is derived from the configuration handed to the restore, not carried inside the file — which means the shape of the cluster you build is the shape the restore adopts. What the restore writes for membership is equally literal:

$ etcdctl member list --write-out=table

--- member list after restore ---
┌──────────────────┬─────────┬─────────┬───────────────────────┬───────────────────────┬────────────┐
│        ID        │ STATUS  │  NAME   │      PEER ADDRS       │     CLIENT ADDRS      │ IS LEARNER │
├──────────────────┼─────────┼─────────┼───────────────────────┼───────────────────────┼────────────┤
│ b71f75320dc06a6c │ started │ default │ http://127.0.0.1:2380 │ http://127.0.0.1:2379 │      false │
└──────────────────┴─────────┴─────────┴───────────────────────┴───────────────────────┴────────────┘

One member, named default, at the peer URL the restore was told about. The snapshot did not decide any of that; the restore parameters did. Scale that to a three-node control plane on replacement hosts and the consequence is plain: the intended membership has to be known before the restore, which means the cluster definition must exist and be authoritative before the state does.

The definition is therefore a recovery artefact in its own right, needing the same custody as a backup: the installer configuration, the node inventory, the CNI manifests and their settings, the storage class definitions with their exact names and parameters, and the distribution version. Rebuild with a storage class named differently from the one the manifests request and every claim in the estate stays unbound — a failure whose root cause is a string, discovered three steps later.

What a re-apply carries that a restore does not

Once the platform exists there are two ways to get the objects back, and they are not equivalent. Re-applying from the repository sends each manifest to the new API server and lets it create the object fresh. Restoring an etcd snapshot writes the previous control plane’s database underneath the new one. Where every object is declared in Git, the re-apply is the better choice, and the reason is visible in what the snapshot restore returned. In the etcd capture, a Deployment that had been scaled to replicas=12 after the snapshot came back at replicas=6, and a ConfigMap key written after the snapshot was simply ABSENT. Nothing failed and no exit code was non-zero; the file faithfully contained an older world. A re-apply from a commit, by contrast, converges the cluster on a state a human reviewed and can read, with a commit id to name it by.

The honest caveat is the objects that are not in the repository: custom resources written by operators, controller-generated state, and anything a person applied from a laptop during a previous incident. If those matter — and in most estates a handful of them do — the snapshot remains the fallback for exactly that subset, and the subset should be known before the incident rather than discovered by absence afterwards.

The data goes in after the claims, not before

With the platform built and the objects applied, the cluster is complete and the service is wrong. The capture shows it by destroying the namespace and rebuilding from the same YAML.

$ kubectl delete namespace rbdr-shop; kubectl apply -f rbdr-shop.yaml

--- destroy the namespace and its volume again ---
namespace fully removed after 0s
pod Ready after 6s
namespace recreated from YAML; volume is empty:
total 8
drwxrwxrwx    2 root     root          4096 Aug 28 14:35 .
drwxr-xr-x    1 root     root          4096 Aug 28 14:35 ..

Six seconds to a Ready Pod, and a volume containing . and ... That is where a full-loss recovery stands at the end of step two, and it is the state teams mistake for the end of the job, because every signal the cluster emits about itself is now green.

The data restore that follows must target the volume the new claim resolves to. That path is not knowable in advance and is not the one in last year’s runbook, so the procedure resolves it at run time from the claim itself.

NS=rbdr-shop
PVC=rbdr-orders
VOL=$(kubectl -n "$NS" get "pvc/$PVC" -o jsonpath='{.spec.volumeName}')
if [ -z "$VOL" ]; then
  echo "claim $PVC is not bound yet; do not start the data restore" >&2
  exit 1
fi
echo "restore target volume: $VOL"

The guard matters as much as the lookup: an unbound claim has no volume name, and a restore that runs anyway writes somewhere useless. With the target resolved, the archive taken before the loss goes back in.

$ tar xf /tmp/rbdr-pv-backup.tar -C $NEWDIR

--- restore the volume contents into the NEW volume directory ---
new PersistentVolume directory: /var/lib/rancher/k3s/storage/pvc-ef5cf541-eebf-4eb3-9c91-cb4e59d6d787_rbdr-shop_rbdr-orders

ORDER-1001,4500.00
ORDER-1002,1250.00
recovered md5 : 9eb4e2ad8e08e1dcaaf87ababab964b0
original md5  : 9eb4e2ad8e08e1dcaaf87ababab964b0
RECOVERED - the application data is back, byte-identical

Both sums read 9eb4e2ad8e08e1dcaaf87ababab964b0, the value recorded when the application first wrote the file. The capture states the conclusion this whole part has been building towards: full recovery needed both the desired state to rebuild the objects and a separate data backup to refill the volume, and neither alone was enough.

The dependencies the new cluster only refers to

A manifest is full of names that resolve outside the cluster, and a rebuilt cluster resolves every one of them again for the first time.

The image registry is the first to bite, because nothing runs without it. A new cluster has new pull credentials, possibly a new network path, and — if the registry lived in the estate that was lost — possibly no images at all.

Secrets held outside the cluster are next. If the workload takes its credentials from an external manager through a syncing controller, then the controller needs its own credential to start, and that credential is the one thing that cannot come from the manager. Certificates behave the same way: the issuer must exist and be trusted before anything it signs is useful, and a rebuilt cluster’s internal certificate authority is new, so anything that pinned the old one has to be updated rather than restored.

DNS and ingress addresses are where the recovery becomes visible outside. The new ingress is reachable at a new address, and until the record customers resolve points at it, a perfectly recovered service is serving nobody. Cloud resources referenced by name — disks, buckets, load balancers, roles — either still exist and must be reattached, or must be recreated under the names the manifests expect.

Ready in 6s, and wrong

The last step decides whether any of the previous four worked, and it is routinely written as the weakest check in the runbook.

Look again at what the cluster reported after the objects were rebuilt and before the data was restored. The apply exited 0, the claim reached Bound, and the Pod reported 1/1 Running and was Ready after 6s. A controller comparing the cluster against the commit would have found nothing left to do. Reading the file the business cares about returned cat: can't open '/data/orders.csv': No such file or directory with exit code 1.

Readiness answers a question about a container: a process started and whatever probe was configured passed, both of which are true of a service whose data is gone. The validation that separates the two states has to touch the data — read a record that existed before the loss and compare it against an independently recorded value, write a record and confirm it persists across a Pod restart, and send one real request through the address the outside world uses rather than a port-forward that bypasses ingress, DNS and certificates.

A transaction that crosses the ingress address, terminates TLS with the new certificate, authenticates with a secret from the external store and reads a row written last week has, in one action, tested all five steps of the sequence.

What to take from this

• Identity follows the cluster you build, not the file you restore. On etcd 3.7.1 the same snapshot restored with matching membership gave cluster id 1c45a069f3a1d796, and with different membership gave 80d54574493dd420.
• A restored data directory came up holding one member named default at http://127.0.0.1:2380, the peer URL the restore was handed. Membership is re-declared by the restore parameters, so the intended topology has to be known before any state goes in.
• Restoring state instead of re-applying it brings the old world back exactly: a Deployment returned at replicas=6 after being scaled to 12, and a key written after the snapshot returned ABSENT, with no error at any point.
• Re-applied manifests rebuilt every object on k3s v1.36.3+k3s1 and the Pod was Ready after 6s, over a volume whose listing was total 8 and the two entries . and ... Object health is not data presence.
• One declared claim named rbdr-orders was backed by pvc-6edd5db0-25f4-4b33-b26b-e2ad78aac9cf, then pvc-a29538af-09d4-4bb8-82bb-b2c222b9db93, then pvc-ef5cf541-eebf-4eb3-9c91-cb4e59d6d787. Resolve the restore target from the bound claim at run time, because only the name was ever in the manifest.
• Only the separate data restore made the service correct: recovered md5 and original md5 both read 9eb4e2ad8e08e1dcaaf87ababab964b0. Both halves were necessary and neither was sufficient.

Cross-course references

• Kubernetes for Production Sysadmins — Part LXXV (Building a Production Cluster) is the definition this lesson’s first step restores from, and Part XCIX (Complete Cluster Loss) works the same scenario from the cluster operator’s side; read them for how the platform is constructed, and this lesson for where that construction sits in the recovery order.
• Git, CI/CD & GitOps for Infrastructure Engineers — Part LXXIV (Reconciliation) and Part LXXXVII (GitOps During Incidents) describe the loop that makes step two so fast and so complete, which is exactly why it is dangerous as a stopping point: reconciliation converges objects on the committed state and has no opinion whatever about the bytes in a volume, which is the gap step three exists to close.
• Secrets, PKI & Certificate Management for Infrastructure Engineers — Part XVIII (Incidents and Recovery) and Part IX (Certificate Lifecycle and Revocation) cover the credential and issuer recovery that step four depends on; a rebuilt cluster has a new internal certificate authority and no bootstrap credential of its own, so a workload can be fully restored by steps one to three and still authenticate nothing.

Quiz