Quorum loss recovery — when a cluster is stuck, snapshot restore is the path

Quorum loss is the disaster-class event every etcd operator trains for but hopes never to run. The cluster is stuck: writes time out, reads may still return data on isolated members but cannot serve consistent state, and no amount of waiting will resolve it. Recovery is to restore the most recent snapshot to a freshly built cluster and validate. This lesson walks the recognition, the sequence, and the validation.

What quorum loss looks like

flowchart LR
    AS[API server] -->|write| E[etcd]
    E -->|quorum reached| OK[commit, status 200]
    E -->|quorum lost| FAIL[timeout, 5xx]
    AS -->|many failures| OP[operator on pager]

The visible signal:

• kubectl apply times out or returns etcdserver: request timed out.
• The API server logs are full of connection refused or context deadline exceeded to the etcd peer URLs.
• etcdctl endpoint status from any surviving member shows the leader as the only responding member, with raft terms incrementing but no commits.
• The cluster’s metric etcd_server_has_leader may flip between 1 and 0 across members.

A secondary signal: any member of the cluster reports “unstarted” or cannot be reached.

$ etcdctl --endpoints=https://10.0.1.10:2379,https://10.0.1.11:2379,https://10.0.1.12:2379 --cacert=... --cert=... --key=... endpoint status --write-out=table

+---------------------------+------------------+---------+---------+
|         ENDPOINT          |        ID        | IS LEADER | ERRORS |
+---------------------------+------------------+---------+---------+
| https://10.0.1.10:2379    |  c5e9a1b2...     |   false  |         |
| https://10.0.1.11:2379    |  d7a1c8f3...     |   false  | timeout |
| https://10.0.1.12:2379    |  e8b4d2a5...     |   false  |         |
+---------------------------+------------------+---------+---------+

All endpoints show IS LEADER = false — there is no elected leader. The cluster is stuck. The API server above it cannot commit; cluster-wide writes are failing.

The decision tree

flowchart LR
    QL[Quorum lost] --> CHECK{Is a quorum of members reachable?}
    CHECK -->|Yes| MAJORITY[Members can talk; election should complete]
    CHECK -->|No| NO_QUORUM[No quorum possible]
    MAJORITY --> WAIT[Wait 30s for next election]
    WAIT -->|still no leader| NO_QUORUM
    NO_QUORUM --> SNAPSHOT[Restore from snapshot]

The two operational states:

• Members can talk but no leader. A transient state during elections; should resolve within the election timeout. If it persists past 30-60 seconds, treat as quorum loss.
• No quorum of members reachable. Persistent. Requires snapshot restore.

The snapshot

The first step in recovery is find the most recent snapshot:

# Find the snapshot files
ls -la /backup/etcd-snapshot-*
# Output:
# -rw------- 1 root root 41289312 Aug 16 02:00 /backup/etcd-snapshot-20260816-0200.db
# -rw------- 1 root root 41289312 Aug 16 03:00 /backup/etcd-snapshot-20260816-0300.db
# -rw------- 1 root root 41289312 Aug 16 04:00 /backup/etcd-snapshot-20260816-0400.db

# Verify the snapshot
etcdutl snapshot status /backup/etcd-snapshot-20260816-0400.db --write-out=table

+----------+----------+------------+----------------+
|   HASH   | REVISION |  TOTAL KEY |   TOTAL SIZE   |
+----------+----------+------------+----------------+
| a1b2c3d4 | 41289312 |       4123 |   82419000     |
+----------+----------+------------+----------------+

A 4-hour-old snapshot on a healthy cluster is reasonable. On a cluster that lost quorum mid-day, the snapshot will be 1-2 hours stale at worst.

The restore sequence

The restore is a controlled rebuild:

1. Stop the API server (and kube-controller-manager and kube-scheduler) on every control-plane host. The API server is the only writer to etcd; it must be off-line.
2. Stop etcd on every host (crictl stop on the static pod or systemctl stop).
3. Move existing data dirs aside. For each member: mv /var/lib/etcd /var/lib/etcd.<timestamp>.
4. Decide where to write the new data. The restore creates new members. Each member’s data is restored from the same snapshot; each member has its own data dir on its own host.

# On the host that will become cp-1:
etcdutl snapshot restore /backup/etcd-snapshot-20260816-0400.db \
  --data-dir=/var/lib/etcd-restore \
  --name=cp-1 \
  --initial-cluster=cp-1=https://10.0.1.10:2380,cp-2=https://10.0.1.11:2380,cp-3=https://10.0.1.12:2380 \
  --initial-advertise-peer-urls=https://10.0.1.10:2380

# Move into place:
mv /var/lib/etcd-restore/* /var/lib/etcd/

5. Repeat for each member, adjusting --name and --initial-advertise-peer-urls.
6. Start etcd on each member. The first member to start becomes a single-member cluster; the second and third members join it.
7. Confirm cluster size = 3 with one leader.
8. Start the API server on each control-plane host.

$ etcdctl --endpoints=https://10.0.1.10:2379 --cacert=... --cert=... --key=... endpoint status --write-out=table

+---------------------------+------------------+---------+---------+
|         ENDPOINT          |        ID        | IS LEADER | ERRORS |
+---------------------------+------------------+---------+---------+
| https://10.0.1.10:2379    |  9f3c6e07...     |   true   |         |
| https://10.0.1.11:2379    |  4b81da55...     |  false   |         |
| https://10.0.1.12:2379    |  2ac7f9e1...     |  false   |         |
+---------------------------+------------------+---------+---------+

Rebuilding on kubeadm

For a kubeadm cluster, the kubeadm-managed static pod manifest needs to be paired with the restore. The clean path:

1. Snapshot the manifest: cp /etc/kubernetes/manifests/etcd.yaml /backup/etcd.yaml
2. Restore the data on each host as in the previous section.
3. Restart the static pod on each host (kubelet observes the manifest; restart is automatic once the data is in place).
4. Wait for the kubelet to start the new member.

A common operator confusion: does the static pod manifest need editing for the restore? Only if the existing manifest has flags that conflict with the restore state. In most cases the manifest is unchanged after a restore because the same --initial-cluster and peer URL flags already match.

The decision to skip state

A snapshot restore rewinds to the snapshot time. Anything created or changed after is gone. The operator has a choice:

• Strict restore — restore exactly what was snapshotted.
• Add-back — restore the snapshot, then re-create things that exist in the lost window.

The second is sometimes necessary. If a Secret was created at 13:00 and the snapshot is from 12:00, the Secret is gone after restore. The operator finds the Secret in some application backup (Vault, GitOps, manual records) and re-creates it after the restore.

The discipline: document what was lost in the restore window and communicate it to consumers of the cluster.

Validating the restored cluster

Before putting the cluster back in production:

1. Member list matches expected: etcdctl member list shows 3 members.
2. One leader: etcdctl endpoint status shows one leader; consistent terms across members.
3. API server can read and write: kubectl get ns returns the namespaces; kubectl run a test pod in default succeeds; delete the test pod.
4. Workload reconstruction: workloads defined in GitOps are reapplied on cluster reconciliation; their replicas rise; pods land on nodes.
5. Backups are resumed: the snapshot cadence resumes.
6. Encryption at rest verified: a new Secret is encrypted at rest (if configured).

flowchart LR
    R[Restore complete] --> V1{member list correct?}
    V1 -->|no| FIX[fix member data dir; restart]
    V1 -->|yes| V2{leader elected?}
    V2 -->|no| WAIT[wait one election interval]
    V2 -->|yes| V3{API server writes succeed?}
    V3 -->|no| APDB[api-server-debug]
    V3 -->|yes| V4{workload reconciles?}
    V4 -->|no| GIT[git-ops debug]
    V4 -->|yes| OK[production-ready]

When not to restore

Sometimes the answer is not restore:

• Snapshot is corrupt or too old. A 2-month-old snapshot on a fast-changing cluster restores state that is so outdated it cannot be reconciled. In this case the answer is to bootstrap a new cluster with fresh state.
• Cluster is recompromised during recovery. An attacker with write access can write ransomware-style state. Restoring from a snapshot they own reinstates the problem. The answer is fresh bootstrap, not restore.

The pre-incident discipline

Every operator whose first response is “restore from snapshot” is learning the procedure during the incident. The pre-incident discipline:

• Run a quarterly restore drill on a separate host. The drill captures the time it takes, the steps that vary, and the mistakes that need runbook updates.
• Test the etcdutl snapshot status command on every snapshot at backup time. A snapshot whose hash does not match is a snapshot that will not restore.
• Document the member peer URLs and --initial-cluster argument. The argument is needed at restore time; finding it in the moment is an avoidable delay.

Quiz

Production discipline

• Treat quorum loss as a disaster. Plan the recovery before the incident; rehearse it quarterly on a non-production host.
• Snapshot before every maintenance. A fresh snapshot bounds the data-loss window of any restore.
• 5 members with 2/2/1 spread is the production target. This is the cheapest way to halve the rate of quorum-loss risk.
• Stop the API server before restoring etcd. The API server is the only writer; if it can write to the new cluster mid-restore, the new cluster’s state is inconsistent.
• Validate the restored cluster before serving production traffic. Member list, leader, API server writes, workload reconciliation.

Quorum loss is rare. Its recovery is a procedure. The production discipline is the procedure, practised.