Runbook: Restore etcd

An etcd restore returns the cluster to the state at the snapshot timestamp. Every write since then is gone. That is the point of the procedure, and it is why a restore is a decision rather than a recovery action: the data-loss window is chosen when the snapshot is chosen.

Which tool restores the snapshot

etcdctl is the network client; etcdutl is the offline utility that operates directly on data files. etcd v3.6.0 removed etcdctl snapshot restore and etcdctl snapshot status in favour of the etcdutl equivalents, and the etcd documentation has shown etcdutl snapshot restore since v3.5. kubeadm 1.34 ships etcd 3.6.5-0, so on a 1.34 cluster etcdutl is the only option.

Taking the snapshot is unchanged: etcdctl snapshot save is a network call against a live member and stays on etcdctl.

# Match the version to the image kubeadm runs on this cluster.
sudo grep -- 'image:' /etc/kubernetes/manifests/etcd.yaml

ETCD_VER=v3.6.5
curl -fsSL "https://github.com/etcd-io/etcd/releases/download/$ETCD_VER/etcd-$ETCD_VER-linux-amd64.tar.gz" -o /tmp/etcd.tar.gz
sudo tar xzf /tmp/etcd.tar.gz -C /usr/local/bin --strip-components=1 --no-same-owner etcd-$ETCD_VER-linux-amd64/etcdutl

etcdutl version

1. Record the cluster facts

Every flag the restore needs is already in the etcd manifest on each node. Read them before anything is stopped, and write them into the change ticket: after step 3 the API server is gone and this is the only copy.

# Run on each control-plane node in turn.
sudo grep -E -- '--(name|initial-advertise-peer-urls|initial-cluster|data-dir)=' \
/etc/kubernetes/manifests/etcd.yaml

# The hostPath backing the data directory.
sudo grep -B 3 -- 'name: etcd-data' /etc/kubernetes/manifests/etcd.yaml

# While the API server is still up, record what the restore must reproduce.
kubectl get nodes -o wide
kubectl get clusterrolebindings,rolebindings -A --no-headers | wc -l
kubectl get pods -A --no-headers | wc -l

kubeadm names each etcd member after its node and derives the peer URL from that node’s advertise address, so a three-node cluster reads back as three distinct names and three distinct URLs. This runbook uses these values throughout; substitute your own.

All three restores share one --initial-cluster string built from that table:

cp-1=https://10.0.1.10:2380,cp-2=https://10.0.1.11:2380,cp-3=https://10.0.1.12:2380

2. Stage and verify the snapshot on every node

The restore is offline and local: each node reads the snapshot from its own filesystem. Copy the same file to all three nodes and verify it on all three before stopping anything.

# Pull and decrypt on one node, then copy the plaintext .db to the others.
restic -r sftp:backup@backup.internal:/srv/etcd restore "$SNAP_ID" --target /tmp/etcd-snapshot
sha256sum /tmp/etcd-snapshot/snap.age | tee /tmp/etcd-snapshot/snap.age.sha256
age -d -i /etc/backup/age-key < /tmp/etcd-snapshot/snap.age > /tmp/etcd-snapshot/snap.db

# Verify on every node that will run the restore.
etcdutl snapshot status /tmp/etcd-snapshot/snap.db -w table
sha256sum /tmp/etcd-snapshot/snap.db

Compare the sha256sum across all three nodes: the restore is only consistent if every member starts from byte-identical input. Compare the reported REVISION against the recovery point in the change ticket. Do not proceed if the hashes differ between nodes, if etcdutl snapshot status reports an error, or if the revision is not the one that was agreed.

A snapshot copied straight out of a data directory rather than taken with etcdctl snapshot save carries no integrity hash and will only restore with --skip-hash-check. If you need that flag, record why in the ticket: it means the snapshot was never verified.

3. Stop the control plane on every node

kubeadm runs the control plane as static Pods, so there are no systemd units to stop and crictl stop alone is useless: the kubelet restarts anything whose manifest is still in the watched directory. Moving the manifests out is what actually stops them.

# Run on every control-plane node.
sudo mkdir -p /etc/kubernetes/manifests.stopped
sudo mv /etc/kubernetes/manifests/kube-apiserver.yaml \
      /etc/kubernetes/manifests/kube-controller-manager.yaml \
      /etc/kubernetes/manifests/kube-scheduler.yaml \
      /etc/kubernetes/manifests/etcd.yaml \
      /etc/kubernetes/manifests.stopped/

# The kubelet removes the Pods within about 30 seconds.
sleep 45
sudo crictl ps --state running | grep -E 'kube-apiserver|kube-scheduler|kube-controller-manager|etcd' \
&& echo 'STILL RUNNING - do not continue' \
|| echo 'all control-plane containers stopped on this node'

Repeat on every control-plane node and confirm the “all stopped” line on each before continuing. This is the step that takes the cluster down.

4. Set the existing etcd data directories aside

etcdutl snapshot restore refuses to write into a data directory that exists and is not empty, which is a useful guard. Moving the live directory aside preserves it for the post-mortem and for the rollback path described at the end of this runbook.

TS=$(date -u +%Y%m%dT%H%M%SZ)

# Run on every control-plane node. Do not delete these directories today.
sudo mv /var/lib/etcd "/var/lib/etcd.broken-$TS"
sudo ls -ld /var/lib/etcd.broken-*
sudo du -sh /var/lib/etcd.broken-*

5. Restore on each node, with that node’s own identity

Each node runs its own command. The snapshot file, the --initial-cluster string and the --initial-cluster-token are the same everywhere; --name and --initial-advertise-peer-urls are that node’s own. The restore writes a fresh data directory at /var/lib/etcd-restore rather than over /var/lib/etcd, matching kubernetes-lxix-04-snapshot-restore-args.

Node cp-1 (10.0.1.10)

sudo etcdutl snapshot restore /tmp/etcd-snapshot/snap.db \
--name cp-1 \
--initial-advertise-peer-urls https://10.0.1.10:2380 \
--initial-cluster cp-1=https://10.0.1.10:2380,cp-2=https://10.0.1.11:2380,cp-3=https://10.0.1.12:2380 \
--initial-cluster-token k8s-restore-20260818 \
--bump-revision 1000000000 \
--mark-compacted \
--data-dir /var/lib/etcd-restore

Node cp-2 (10.0.1.11)

sudo etcdutl snapshot restore /tmp/etcd-snapshot/snap.db \
--name cp-2 \
--initial-advertise-peer-urls https://10.0.1.11:2380 \
--initial-cluster cp-1=https://10.0.1.10:2380,cp-2=https://10.0.1.11:2380,cp-3=https://10.0.1.12:2380 \
--initial-cluster-token k8s-restore-20260818 \
--bump-revision 1000000000 \
--mark-compacted \
--data-dir /var/lib/etcd-restore

Node cp-3 (10.0.1.12)

sudo etcdutl snapshot restore /tmp/etcd-snapshot/snap.db \
--name cp-3 \
--initial-advertise-peer-urls https://10.0.1.12:2380 \
--initial-cluster cp-1=https://10.0.1.10:2380,cp-2=https://10.0.1.11:2380,cp-3=https://10.0.1.12:2380 \
--initial-cluster-token k8s-restore-20260818 \
--bump-revision 1000000000 \
--mark-compacted \
--data-dir /var/lib/etcd-restore

What each flag is doing:

Confirm on each node that the restore produced a data directory and that its ownership matches the directory kubeadm created. kubeadm sets no runAsUser on the etcd static Pod, so the container runs as root and a root-owned directory is what it expects; running etcdutl under sudo gives that.

sudo ls -l /var/lib/etcd-restore/member/
sudo stat -c '%U:%G %a %n' /var/lib/etcd-restore /var/lib/etcd.broken-*

6. Repoint the etcd manifest at the restored data directory

kubeadm mounts the host data directory into the container at the same path and passes that path as --data-dir, so changing the etcd-data hostPath alone is enough: the container still sees /var/lib/etcd, now backed by the restored directory. Leave --data-dir and the volumeMount alone.

# The manifest is still staged out of the watched directory from step 3.
sudo cp /etc/kubernetes/manifests.stopped/etcd.yaml /root/etcd.yaml.pre-restore

sudo sed -i 's|path: /var/lib/etcd$|path: /var/lib/etcd-restore|' \
/etc/kubernetes/manifests.stopped/etcd.yaml

# Exactly one line should differ, and it must be the etcd-data volume.
sudo diff /root/etcd.yaml.pre-restore /etc/kubernetes/manifests.stopped/etcd.yaml
sudo grep -B 3 -- 'name: etcd-data' /etc/kubernetes/manifests.stopped/etcd.yaml

If diff reports more than the single path: line, or if the changed line sits under etcd-certs instead of etcd-data, restore /root/etcd.yaml.pre-restore and edit by hand.

The alternative is to move the restored directory into place with mv /var/lib/etcd-restore /var/lib/etcd and leave the manifest untouched, as kubernetes-lxix-04-snapshot-restore-args shows. Both reach the same state. Repointing the hostPath keeps the restored and pre-restore directories visibly distinct on disk, which is why it is used here; whichever you pick, use it on all three nodes.

7. Start etcd, and only etcd

Return etcd.yaml alone. The three members must form a cluster before any API server is allowed to write to it.

# Run on every control-plane node, then wait.
sudo mv /etc/kubernetes/manifests.stopped/etcd.yaml /etc/kubernetes/manifests/

sleep 60
sudo crictl ps --name etcd
sudo crictl logs "$(sudo crictl ps -q --name etcd)" 2>&1 | tail -30

Expect each member to log that it has published its peer URL and that a leader has been elected. A member logging a cluster ID mismatch was restored with a different --initial-cluster or --initial-cluster-token than its peers; re-run step 5 on that node alone with the recorded values.

8. Verify the restored etcd cluster

Run these from any control-plane node. etcdctl is the right tool here: these are live API calls, not data-file operations.

export ETCDCTL_ENDPOINTS=https://10.0.1.10:2379,https://10.0.1.11:2379,https://10.0.1.12:2379
export ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt
export ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt
export ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key

sudo -E etcdctl member list -w table
sudo -E etcdctl endpoint health --cluster -w table
sudo -E etcdctl endpoint status --cluster -w table
sudo -E etcdctl alarm list

Four things must hold before step 9:

• member list shows exactly three members, with the three names and the three peer URLs from the table in step 1, and no fourth member.
• endpoint health reports every endpoint healthy.
• endpoint status shows exactly one member with IS LEADER true, and the same RAFT TERM and revision on all three.
• alarm list is empty. A NOSPACE alarm here means the restored database is already over the quota, and the API server will fail to write.

Do not continue while any member is missing, unhealthy or reporting a different revision. Starting the API servers against a half-formed cluster turns a recoverable state into a second incident.

9. Start the rest of the control plane

sudo mv /etc/kubernetes/manifests.stopped/kube-apiserver.yaml \
      /etc/kubernetes/manifests.stopped/kube-controller-manager.yaml \
      /etc/kubernetes/manifests.stopped/kube-scheduler.yaml \
      /etc/kubernetes/manifests/

sleep 60
sudo crictl ps --state running | grep -E 'kube-apiserver|kube-scheduler|kube-controller-manager|etcd'
sudo rmdir /etc/kubernetes/manifests.stopped

rmdir refuses to remove the staging directory unless it is empty, which is a cheap check that no manifest was left behind.

10. Verify the API server

kubectl get --raw /livez?verbose
kubectl get --raw /readyz?verbose

kubectl get nodes -o wide
kubectl get pods -A --no-headers | wc -l
kubectl get clusterrolebindings,rolebindings -A --no-headers | wc -l
kubectl auth can-i get pods

Compare the counts against the values recorded in step 1. They should match the cluster as it was at the snapshot revision, not as it was immediately before the restore; a difference is the data-loss window made visible, and it belongs in the ticket.

11. Verify add-ons and workloads

kubectl -n kube-system get pods -l k8s-app=kube-dns
kubectl -n kube-system get pods -l k8s-app=kube-proxy
kubectl -n kube-system get daemonset

# DNS and scheduling both work end to end.
kubectl run restore-check --image=registry.k8s.io/pause:3.10.1 --restart=Never
kubectl wait --for=condition=Ready pod/restore-check --timeout=60s
kubectl delete pod restore-check

# Spot-check application workloads.
kubectl get deploy,sts -A
kubectl describe deploy/web -n prod | head -30
curl -fsS https://app.prod.example/healthz

12. Record the restore

Write the record while the detail is fresh, and keep the disposition of the broken data directories in it — they are the rollback path until someone deletes them.

# etcd restore record

Change ticket:        <ticket>
Restored by:          <operator>
Approved by:          <approver>
Started / finished:   <UTC> / <UTC>

Snapshot file:        <path>
Snapshot sha256:      <hash>
Snapshot revision:    <revision from etcdutl snapshot status>
Snapshot taken at:    <UTC>
Data-loss window:     <snapshot timestamp> to <incident timestamp>

Restore parameters:
  initial-cluster:        <the shared string>
  initial-cluster-token:  <token>
  bump-revision:          <n>
  restored data dir:      /var/lib/etcd-restore
  preserved data dirs:    /var/lib/etcd.broken-<timestamp> on each node

Post-restore verification:
- [ ] member list shows three members with the expected names and peer URLs
- [ ] endpoint health healthy on all three
- [ ] endpoint status: one leader, matching revision
- [ ] alarm list empty
- [ ] /readyz all checks ok
- [ ] every node Ready
- [ ] object counts reconciled against the pre-restore capture
- [ ] workload smoke tests pass

Follow-up:
- [ ] Post-mortem scheduled
- [ ] Root cause identified
- [ ] Backup validation pipeline exercised against this snapshot
- [ ] /var/lib/etcd.broken-<timestamp> deleted on <date> by <operator>

Common pitfalls

An etcd restore is a controlled detonation. The recovery point is chosen deliberately, every node restores with its own identity and the same membership, the old data directories survive until someone decides otherwise, and the restored state is validated through etcd and through the API server before a single workload is readmitted.