Recover Kubernetes cluster state

1 · Prerequisites

Confirm every item is in place before any state change.

• A named incident owner, and agreement that what is being recovered is the cluster's record of desired state rather than any application's data
• Every snapshot you hold, with the status output and checksum recorded when each was written, and the wall-clock time each was taken
• Root or equivalent on every control-plane node, and a way to stop the API server and the datastore on all of them and keep them stopped
• The etcd and etcdutl binaries the cluster actually runs, present on the host performing the restore, with their versions read rather than assumed
• The membership the cluster is to have after the restore, written down: every member name, every peer URL, and the cluster token
• A list, however rough, of what changed in the hours before the incident, so the reversion this restore causes can be predicted rather than discovered by a user
• A change record and somewhere to write timings, because the elapsed time of this procedure is the recovery time this estate actually has

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

• · Read the versions of the binaries you are about to use. etcd --version and etcdutl version on the host that will perform the restore. On etcd 3.7.1 the snapshot subcommands live in etcdutl, and a runbook naming the wrong binary spends its first minutes proving that the tooling is not broken.
• · List every snapshot you have, with its age — not only the newest. Recovery point selection needs candidates. A procedure that says "restore the latest" has made the choice on your behalf and hidden what it cost.
• · **Verify the candidate snapshot with etcdutl snapshot status --write-out=table** and compare HASH, REVISION and TOTAL KEYS against the values recorded when the file was written. Those three values are the entire self-description of an etcd snapshot; there is no manifest and no record of which cluster produced it.
• · Confirm the checksum recorded at backup time still matches the file. The status output tells you what the file says about itself now. The checksum is what lets you say it is the file you took.
• · Establish the intended membership on paper before starting: every member name, every peer URL, and the cluster token. This is an input to the restore, not something the restore works out for you.
• · Confirm you can stop the API server and the datastore on every control-plane node, and that nothing will restart them while the restore runs. A member that comes back early rejoins with a history the others cannot reconcile.
• · Confirm the exposure of the snapshot file has been considered. Unless encryption at rest is configured on the API server, it holds every Secret in readable form: the capture behind this runbook read the value S3cretValue back out of the restored cluster unchanged. Copy and store the file accordingly.
• · Record the start time, the candidate snapshot and its recorded identity in the incident record before running anything that writes.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1. 1Select the recovery point explicitly, and write down what it costs. Choose from the listed candidates, then name every change made after the moment that snapshot was taken. The recovery point of this recovery is the age of the file you pick; the restore itself will report nothing about what falls inside that window.
2. 2Decide between restoring the datastore and rebuilding the cluster. Restore when the datastore is lost or corrupt and the workload definitions are not reproducible from a source of truth. Rebuild and re-apply manifests when a current source of truth covers every namespace that matters, because that path has no reversion window at all. Record the decision and its reason before you act on it.
3. 3Stop the API server and the datastore on every member before touching any data directory. A restore rewrites raft history, so a member restored while its peers still serve holds a history the others cannot reconcile. The disagreement surfaces later as members refusing to join, not as an error at restore time.
4. 4Move the existing data directories aside rather than deleting them, to an rbdr- prefixed path on the same host. They are the only evidence of the pre-restore state and the only material an escalation can work from.
5. 5Distribute one snapshot file to every member and prove it arrived intact, comparing the checksum on each host against the value recorded at backup time. Two members restored from two different files is the failure this step exists to prevent, and it cannot be detected from the datastore afterwards.
6. 6Restore on every member from that same file, into a new data directory. On etcd 3.7.1 this is etcdutl snapshot restore; the etcdctl form printed only the parent command's usage text. Run it separately on each member — the command writes a data directory, it does not talk to a cluster.
7. 7Pass the full intended membership at restore time, identical on every member apart from that member's own name and peer URL. The cluster id is derived from the membership configuration passed here rather than carried inside the file: the capture restored the same snapshot twice and got 1c45a069f3a1d796 with identical parameters and 80d54574493dd420 with different ones.
8. 8Start the members together and confirm they agree before going any further. Every member must report the same cluster id and the same membership list. A member that disagrees is not a member to correct in place; it is an abort.
9. 9Start the API server and confirm the datastore answers, then confirm the nodes register. This proves the platform is running. It proves nothing yet about the workloads, and it is not the point at which anybody may be told the cluster is recovered.
10. 10Read back what returned, in the units the business uses. For every object on the change list from step 1, compare the live specification against what it was before the incident. In the capture the Deployment returned at replicas=6 rather than replicas=12, and a ConfigMap key written after the snapshot was simply absent — both with exit code 0.
11. 11Close the reversion window deliberately. Re-apply the manifests from the source of truth, or re-make each change by hand from the list, and record for every item whether it was re-applied or knowingly accepted. A restored cluster left alone is a cluster quietly running an older desired state.
12. 12Validate the service rather than the process. Have the owner of the workload exercise it end to end. Whether the application's own data is present and correct is a different question with its own runbook; the datastore holds desired state, not the contents of the volumes the workloads write to.
13. 13Record the snapshot identity, both cluster ids, the membership passed and the timings before standing down. The interval from decision to validated service is the recovery time this estate has, as distinct from the one in the plan.

4 · Verification

Confirm the procedure actually fixed the problem.

• ✓Every member reports the same cluster id, and the membership list read from each member shows the same members with the expected peer URLs.
• ✓The checksum of the restore source file recorded on each host is identical, proving every member was restored from one snapshot.
• ✓The datastore's current revision is at or above the REVISION the chosen snapshot reported, and the key count is consistent with its TOTAL KEYS.
• ✓The API server serves requests and every node registers. This is the platform check; on its own it is not evidence of recovery.
• ✓Every object on the post-snapshot change list has been compared against its pre-incident value, and each difference is recorded as either re-applied or knowingly accepted.
• ✓Controllers have reconciled: workloads report the replica counts the manifests ask for rather than the counts the snapshot held.
• ✓The service the cluster exists to run answers a real request end to end, exercised by the person who owns it and not by the person who ran the restore.
• ✓Elapsed time from the recovery-point decision to the validated service is recorded and compared against the objective the service owner was given.

5 · Rollback

If verification fails, undo the procedure in reverse order.

• ↶The data directories moved aside before the restore are the rollback. Leave them in place, named and owned, until the restored cluster has been validated and has run long enough to trust.
• ↶If the wrong snapshot was restored, do not correct one member and leave the others. Stop every member again and repeat the whole sequence from the correct file, on all of them.
• ↶If a member was started with a membership different from its peers, wipe that member's restored data directory and redo it from the same file with the same membership. Adding and removing members until the running cluster agrees is not a rollback; it is a second incident.
• ↶Undo the mutations made by reflex during the incident: alert silences, paused GitOps sync, scaled-down controllers, edited probes, temporary RBAC. List them at the end and revert each one explicitly — an unrecorded silence outlives the incident that justified it.
• ↶If workloads were re-applied from a source of truth that was itself behind, fix the source of truth and reconcile again rather than hand-editing live objects, or the cluster and the repository disagree from this moment onward.
• ↶Record what was discarded. A restore is a decision about which version of the estate survives, and that decision needs a written owner.

6 · Escalation

When the runbook isn't enough, contact:

• · Members disagree after the restore — different cluster ids, or different membership lists. Stop. Do not add or remove members to force agreement; escalate to the platform owner with the exact restore parameters used on each host.
• · A member turns out to have been restored from a different snapshot. Abort, escalate, and restart the sequence from stopped members and one verified file.
• · The chosen snapshot's HASH, REVISION or TOTAL KEYS disagree with what was recorded when it was written. Escalate before restoring anything: a file that cannot be identified is not a recovery point.
• · Every snapshot you hold post-dates the event you are recovering from. Escalate to the data owner. Choosing between a corrupted recent state and a much older clean one is not a decision the responder makes alone at three in the morning.
• · Elapsed time is approaching the objective the service owner was given. Escalate to the incident owner with the measured remaining work, so that partial service can be considered rather than waited for.
• · The snapshot file has left its usual custody — copied to a laptop, a file share, or a ticket attachment. Escalate to security: unless encryption at rest is configured, it carries every Secret in the cluster in readable form.

References

1. etcd documentation, Disaster recovery
2. etcdutl command reference, etcd-io/etcd
3. Kubernetes documentation, Operating etcd clusters for Kubernetes
4. etcd documentation, Frequently asked questions