etcd snapshot restore — the cluster's ultimate recovery

A snapshot restore is the last thing you do to a cluster, and it is destructive: every object created since the snapshot was taken disappears, so a six-hour-old backup costs six hours of Deployments, Secrets, and PVC bindings. The procedure works only if the control plane and etcd are stopped before the data directory is replaced, and it works at all only if somebody has previously loaded that snapshot into a lab cluster and watched it come up. A backup nobody has restored is a file, not a recovery plan.

The snapshot

The etcd snapshot is the cluster’s state at a point in time. The snapshot is saved as a file (e.g., /var/backups/etcd/snapshot.db). The snapshot is verified by loading it into a lab cluster.

flowchart LR
    A[Cluster] --> B[Snapshot]
    B --> C[Backup storage]
    C --> D[Lab cluster]
    D --> E[Verification]

The snapshot is the cluster’s last resort.

The verification

The snapshot is verified by loading it into a lab cluster:

# Restore the snapshot to a lab cluster
etcdutl snapshot restore /var/backups/etcd/snapshot.db \
  --data-dir /var/lib/etcd-lab

# Start the lab cluster
etcd --data-dir /var/lib/etcd-lab \
  --listen-client-urls http://0.0.0.0:2379 \
  --advertise-client-urls http://localhost:2379

# Verify the snapshot
ETCDCTL_API=3 etcdctl --endpoints=http://localhost:2379 get /registry --prefix --keys-only | head -20

The verification is the cluster’s preparedness.

The diagnostic

The canonical diagnostic:

# 1. Check the snapshot
ls -la /var/backups/etcd/

# 2. Verify the snapshot
etcdutl snapshot status /var/backups/etcd/snapshot.db

# 3. Check the cluster's state
ETCDCTL_API=3 etcdctl endpoint health

# 4. Check the API server
kubectl get --raw /healthz

# 5. Check the kubelet
journalctl -u kubelet -n 200

The diagnostic is the snapshot, the cluster’s state, and the API server.

The restoration

The canonical restoration:

# 1. Stop the control plane and etcd on every control-plane node.
#    There is no systemctl step here: kubeadm runs all four as static
#    Pods, and the kubelet stops a static Pod when its manifest leaves
#    /etc/kubernetes/manifests/.
mkdir -p /root/manifests-parked
mv /etc/kubernetes/manifests/kube-apiserver.yaml \
   /etc/kubernetes/manifests/kube-controller-manager.yaml \
   /etc/kubernetes/manifests/kube-scheduler.yaml \
   /etc/kubernetes/manifests/etcd.yaml \
   /root/manifests-parked/

# 2. Confirm they are gone. kubectl no longer answers, so ask the
#    container runtime on the node.
crictl ps | grep -E 'kube-apiserver|kube-controller-manager|kube-scheduler|etcd'
# Expected: no rows, within one kubelet fileCheckFrequency (20s default)

# 3. Restore the snapshot into a fresh directory, on every
#    control-plane node, from the same snapshot file. --name is that
#    node's etcd member name, which kubeadm sets to the node name.
etcdutl snapshot restore /var/backups/etcd/snapshot.db \
  --data-dir /var/lib/etcd-restore \
  --name cp-1 \
  --initial-cluster cp-1=https://10.0.1.10:2380,cp-2=https://10.0.1.11:2380,cp-3=https://10.0.1.12:2380 \
  --initial-advertise-peer-urls https://10.0.1.10:2380

# 4. Replace the etcd data directory, keeping the old one for rollback
mv /var/lib/etcd /var/lib/etcd-old
mv /var/lib/etcd-restore /var/lib/etcd

# 5. Start etcd first and let the members form a quorum
mv /root/manifests-parked/etcd.yaml /etc/kubernetes/manifests/
crictl ps | grep etcd

# 6. Start the rest of the control plane
mv /root/manifests-parked/kube-apiserver.yaml \
   /root/manifests-parked/kube-controller-manager.yaml \
   /root/manifests-parked/kube-scheduler.yaml \
   /etc/kubernetes/manifests/

# 7. Verify the cluster
kubectl get nodes
kubectl get pods -A

The restoration is the cluster’s recovery.

Why kubectl delete pod does not restart a static Pod

The reflex on an ordinary Pod is kubectl delete pod and let a controller recreate it. That does not work here, and it is worth knowing before an outage rather than during one.

What kubectl get pods -n kube-system shows for etcd-cp-1 is a mirror Pod: a read-only API object the kubelet publishes so the API server can describe a Pod that is actually driven by a local file. Deleting the mirror Pod deletes the API object. The kubelet still has the manifest, notices the mirror is missing, and re-publishes it. The etcd container is never stopped and the etcd process never restarts.

Three things do restart a static Pod, and all three act on the file rather than on the API:

• Move the manifest out of /etc/kubernetes/manifests/ and back. Use this when you need a controlled stop, because the component stays down for exactly as long as the file is out of the directory.
• Edit the manifest in place. The kubelet re-reads the directory every fileCheckFrequency (20s by default) and recreates the Pod when a file’s content changes. This is what makes an added or changed flag take effect.
• systemctl restart kubelet. A real unit, but it restarts the supervisor, which re-reads every manifest and brings all the static Pods back. Use it when the kubelet is the problem, not as a way to restart one component.

The remediation

The remediation depends on the failure:

# Option 1: restore from the snapshot (the full procedure above)
etcdutl snapshot restore /var/backups/etcd/snapshot.db \
  --data-dir /var/lib/etcd-restore

# Option 2: put an already-restored data directory into place.
#   Replace the contents of the directory the manifest already points
#   at rather than editing --data-dir: kubeadm mounts /var/lib/etcd
#   into the container as a hostPath volume, so the flag and the
#   volume would both have to change.
mv /var/lib/etcd /var/lib/etcd-old
mv /var/lib/etcd-restore /var/lib/etcd

# Either way, the restart is a manifest move, not a service restart
mv /etc/kubernetes/manifests/etcd.yaml /root/manifests-parked/
mv /root/manifests-parked/etcd.yaml /etc/kubernetes/manifests/

The remediation is the snapshot restore.

Production discipline

A snapshot restore is the cluster’s hypothesis. The discipline is to walk the 11-step methodology applied to the etcd, identify the cause, apply the remediation. The etcd is the cluster’s state; the remediation is the snapshot restore.

• Verify the snapshot before you need it. A snapshot is verified by restoring it into a lab cluster and reading /registry, not by checking that the file exists.
• Rehearse the restoration. The stop is a manifest move and the restore runs on every control-plane node from the same snapshot file; neither is a step to work out during an outage.

Quiz