Etcd recovery scenarios — snapshot, restore, and the quorum trap

Etcd is the cluster’s source of truth. Recovery from etcd loss is the most consequential DR operation. This lesson walks etcd snapshot creation with etcdctl snapshot save, the restore procedure, the quorum trap, and the operational discipline.

Etcd snapshot creation

etcdctl snapshot save /var/backups/etcd-snapshot.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/peer.crt \
  --key=/etc/kubernetes/pki/etcd/peer.key

The flags:

• --endpoints — the etcd endpoint. For a local snapshot, the loopback. For a remote snapshot, the peer URL of a healthy member.
• --cacert, --cert, --key — the TLS credentials. These are the same credentials etcd uses for peer communication.

The snapshot file is a portable, self-contained etcd data file. It can be moved to any host and restored with etcdutl snapshot restore.

etcdutl snapshot status /var/backups/etcd-snapshot.db -w table

+----------+----------+------------+------------+
|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| 5f8a1b   |   12345  |     1247   |     4.2 MB |
+----------+----------+------------+------------+

The restore procedure

# 1. Stop etcd on all control-plane nodes. kubeadm runs etcd as a
#    static Pod, so there is no etcd.service: the kubelet stops the
#    Pod when its manifest leaves /etc/kubernetes/manifests/.
mkdir -p /root/manifests-parked
mv /etc/kubernetes/manifests/etcd.yaml /root/manifests-parked/
crictl ps | grep etcd
# Expected: no rows, within one kubelet fileCheckFrequency (20s default)

# 2. Restore on the first node
etcdutl snapshot restore /var/backups/etcd-snapshot.db \
  --data-dir=/var/lib/etcd-restore \
  --name=cp-1 \
  --initial-cluster=cp-1=https://10.0.0.1:2380,cp-2=https://10.0.0.2:2380,cp-3=https://10.0.0.3:2380 \
  --initial-advertise-peer-urls=https://10.0.0.1:2380

# 3. Move the restored data dir into place. Move the old directory
#    aside rather than copying into it — a leftover member/ directory
#    from the previous cluster is what makes the member refuse to join.
mv /var/lib/etcd /var/lib/etcd.broken
mv /var/lib/etcd-restore /var/lib/etcd

# 4. Start etcd again by putting the manifest back
mv /root/manifests-parked/etcd.yaml /etc/kubernetes/manifests/
crictl ps | grep etcd
# Expected: an etcd container with a new ID, within ~20s

# 5. Read its log from the node while kubectl is still unavailable
crictl ps -a | grep etcd     # the newest etcd container is the restored one
ETCD_CONTAINER=$(crictl ps -a --name etcd -q | head -n 1)
crictl logs "$ETCD_CONTAINER"

The flags:

• --data-dir — the destination directory for the restored data. This must be on a dedicated disk with enough space.
• --name — the member name. Must match the --initial-cluster and --initial-advertise-peer-urls.
• --initial-cluster — the list of all members in the cluster, with their peer URLs. Used at bootstrap.
• --initial-advertise-peer-urls — this member’s peer URL.

The restore creates a new data directory from the snapshot. The etcd member comes up reading the restored data.

The quorum trap

flowchart TD
    A[3-member etcd] --> B{How many alive?}
    B -->|3| C[Healthy]
    B -->|2| D["Quorum lost, read-only"]
    B -->|1| E["Cluster refused, single point"]
    D --> F{Restore from snapshot?}
    F -->|Yes| G[Restore to new cluster]
    F -->|No| H[--force-new-cluster on survivor]

The quorum trap occurs when a majority of etcd members are dead:

• 3 of 3 dead. Cluster is gone. Recovery requires etcdutl snapshot restore to a new cluster.
• 2 of 3 dead. Quorum lost; etcd refuses writes. Recovery requires either:
  • --force-new-cluster on the surviving member to bootstrap a single-member cluster from its existing data (data loss possible for uncommitted writes), or
  • etcdutl snapshot restore to provision new members (data loss = snapshot interval).
• 1 of 3 dead. Quorum is intact (2 of 3 is a majority). The dead member is replaced; etcd rebalances. No data loss.

Runtime configuration

The etcd configuration is in /etc/etcd/etcd.conf.yaml (or the kubeadm-managed /etc/kubernetes/manifests/etcd.yaml for static-pod etcd):

data-dir: /var/lib/etcd
listen-client-urls: https://10.0.0.1:2379
advertise-client-urls: https://10.0.0.1:2379
listen-peer-urls: https://10.0.0.1:2380
initial-advertise-peer-urls: https://10.0.0.1:2380
initial-cluster: cp-1=https://10.0.0.1:2380,cp-2=https://10.0.0.2:2380,cp-3=https://10.0.0.3:2380
initial-cluster-token: etcd-cluster-1
cert-file: /etc/kubernetes/pki/etcd/server.crt
key-file: /etc/kubernetes/pki/etcd/server.key
client-cert-auth: true
trusted-ca-file: /etc/kubernetes/pki/etcd/ca.crt
peer-cert-file: /etc/kubernetes/pki/etcd/peer.crt
peer-key-file: /etc/kubernetes/pki/etcd/peer.key
peer-client-cert-auth: true
peer-trusted-ca-file: /etc/kubernetes/pki/etcd/ca.crt

The configuration determines how etcd listens for client traffic (API server) and peer traffic (other members). The restore procedure must use the same configuration to ensure the new cluster’s identity matches the original.

The operational failure modes

Etcd recovery fails in production for predictable reasons:

• Snapshot is corrupt. The snapshot file is corrupted in transit or storage. etcdctl snapshot status reports the corruption; restore fails.
• TLS certificates expired. The etcd peer and client certificates have expired during the recovery window. Members cannot authenticate to each other.
• Data directory on the wrong disk. The restored data directory is on a disk that is too small or is shared with another workload. etcd starts but fails on first write.
• Initial cluster wrong. The --initial-cluster flag points to the wrong peer URLs or has the wrong member names. etcd refuses to start.
• Snapshot predates certificates. The snapshot was taken before a certificate rotation; the restored etcd uses the old certificates, which the API server no longer trusts.

Quiz

Production discipline

Etcd recovery in production rests on five non-negotiable elements:

• Snapshot on a schedule. Hourly or daily, depending on RPO. The cadence must be ≤ RPO.
• Snapshot off-cluster. S3, NFS, tape. A snapshot on the same machine as etcd is not a backup.
• Test restore quarterly. The procedure that has never been executed takes longer than the RTO.
• Document both recovery paths. Both —force-new-cluster (data preservation) and snapshot restore (safety) must be in the runbook.
• Rehearse the quorum trap. The 2-of-3 scenario is the most operationally subtle. Rehearse it.

Etcd recovery is the most consequential DR event. A runbook that has never been tested is not a runbook. The discipline is to test the procedure before the disaster, not during it.