When to restore — the decision tree

A restore is destructive: it rewinds the cluster’s state to a snapshot. Done unnecessarily, it is the cause of outage. Done too late, it is the cause of unnecessary data loss. This lesson walks the decision tree that determines whether a restore is the right action, the prerequisites that make it safe, and the communication discipline that keeps stakeholders aligned.

The decision tree

flowchart LR
    A[Symptom] --> Q1{Cluster still serves requests?}
    Q1 -->|No| Q2{Recent healthy snapshot exists?}
    Q1 -->|Yes| Q3{State is corrupted?}
    Q2 -->|Yes| R[Restore from snapshot]
    Q2 -->|No| NR1[Bootstrap fresh cluster]
    Q3 -->|Yes| R
    Q3 -->|No| NR2[Investigate; no restore]

The questions lead to one of three answers:

• Restore — the cluster is broken or the state is wrong, and a usable snapshot exists.
• Bootstrap fresh — no usable snapshot, or the snapshot is too stale.
• Investigate — the issue is not state-related; no restore is needed.

Recognising that restore is the answer

The failure modes that warrant restore:

The unifying property: the etcd state is wrong or inaccessible, and an alternative recovery path exists (a snapshot from before the failure).

Recognising that restore is not the answer

Some failures do not need a restore:

The unifying property: the failure is recoverable without rewriting the cluster’s state.

Pre-restore validation of the snapshot

Before restoring, validate the snapshot:

# 1. The file exists and has the expected size
ls -la /backup/etcd-snapshot.db

# 2. The hash matches the upload verification
sha256sum /backup/etcd-snapshot.db

# 3. The status is non-zero
etcdutl snapshot status /backup/etcd-snapshot.db --write-out=table

+----------+----------+------------+----------------+
|   HASH   | REVISION |  TOTAL KEY |   TOTAL SIZE   |
+----------+----------+------------+----------------+
| a1b2c3d4 | 41289312 |       4123 |   82419000     |
+----------+----------+------------+----------------+

A zero or unparseable hash means the snapshot is corrupt; do not restore from it. A revision older than the cluster’s known state points to a stale snapshot; reconsider.

The state capture before restore

Before the restore, capture the cluster’s current state. This is the documentation of “what we are losing”:

# Capture every API object as YAML
kubectl get all,cm,secret,sa,role,rolebinding,clusterrole,clusterrolebinding,\
  crd,cr -A -o yaml > /backup/pre-restore-state.yaml

# Capture namespace list
kubectl get namespaces -o yaml > /backup/pre-restore-namespaces.yaml

# Capture events (short-retention)
kubectl get events -A > /backup/pre-restore-events.txt

# Capture lease objects (token controllers, leader election)
kubectl get leases -A -o yaml > /backup/pre-restore-leases.yaml

The pre-restore state capture is the recovery of last resort: if a critical object was lost in the restore and the recovery is straightforward, the captured state has its YAML. Some objects (leases, events) are not recoverable; document their loss.

The communication

Before the restore, communicate:

• What is being done. Restore from snapshot dated X.
• Why. Cluster has lost quorum / state is corrupt / DR.
• Expected duration. ~30-60 minutes for the restore; another 30-60 minutes for validation; 1-2 hours for workloads to reconcile.
• What will be lost. State from snapshot time to now.
• What will work after restore. Most workloads will resume from GitOps; some Secrets / ConfigMaps / nodes may need manual recovery.
• Who has approved. Two-person approval is the standard for a destructive action.

A restore is a change. It is not an automatic recovery; it is a decision. Document the decision in the incident log.

flowchart LR
    A[Decision to restore] --> B[Communicate to stakeholders]
    B --> C[Pre-restore state capture]
    C --> D[Snapshot validation]
    D --> E[Restore execution]
    E --> F[Validation]
    F --> G[Communication: restore complete]

The “I might need a restore” trap

An operator who is uncertain about whether to restore should:

1. Capture the current state (so a future restore has a starting point).
2. Snapshot the current cluster (so the latest state is preserved even if restore goes wrong).
3. Document the symptoms with evidence.
4. Escalate to a senior operator or incident commander.

The decision to restore is rarely reversible; the uncertainty must be resolved before pulling the trigger.

The post-decision checklist

Once the decision to restore is made, the operator has a checklist:

The total runtime: ~2-3 hours for a healthy restore; more if the validation surfaces issues.

When the decision tree says “no restore”

Not every symptom leads to restore. The decision tree returns “no restore” for:

• Transient etcd slowness (no data corruption).
• Single member failures (quorum preserved).
• Member caught up and rejoined.
• Bad controller logic that a workload fix can address.
• Upstream issues (DNS, network) where the etcd is healthy.

The “no restore” answer comes with its own follow-up: investigate, resolve, document.

Quiz

Production discipline

• Apply the decision tree. Restore is one of three options (along with “investigate” and “bootstrap fresh”); choose deliberately.
• Capture pre-restore state. Always; the cost is minutes; the benefit is recovery of last resort.
• Validate the snapshot. Hash, status, freshness.
• Communicate. Destructive actions need stakeholder awareness; restore is destructive.
• Document the decision. The incident log captures who decided what, when, why.

Restore is a high-impact operation; the decision is the operator’s. The discipline of the decision tree is what makes the right answer consistent.