etcd quorum loss

Reported symptoms

At 14:07 the platform on-call is paged by a deploy pipeline, not by monitoring.

Every kubectl apply hangs for about seven seconds and returns etcdserver: request timed out. kubectl scale does the same. kubectl get answers immediately and looks normal.

Nothing customer-facing is wrong: running Pods serve traffic, DNS answers, and every service dashboard is green. The only application symptom anyone can find is a rollout started at 14:02 stuck at 4 of 12 replicas with no new events, and an HPA that has stopped reacting to a visibly growing queue.

The odd detail arrives when someone checks the nodes. Every node reports Ready, and every node condition carries a LastHeartbeatTime of 14:07 - still 14:07 four minutes later, on every node in the cluster.

There is a change record open. The virtualisation team is applying a firmware update to a single hypervisor, esx-14. The record says: single host, no service impact, all guests are HA-protected and will restart elsewhere.

The HA design review for this cluster, approved two years ago, specified three control-plane nodes in three separate failure domains. The three nodes carry three distinct topology.kubernetes.io/zone labels, and they always have.

Evidence provided

$ kubectl -n kube-system exec etcd-cp-1 -- etcdctl --cacert=... --cert=... --key=... --endpoints=https://10.30.0.11:2379,https://10.30.0.12:2379,https://10.30.0.13:2379 endpoint status -w table

+--------------------------+------------------+-----------+------------------------+
|         ENDPOINT         |        ID        | IS LEADER |         ERRORS         |
+--------------------------+------------------+-----------+------------------------+
| https://10.30.0.11:2379  | 3a1f0b7c9d2e4a55 |   false   | etcdserver: no leader  |
| https://10.30.0.12:2379  |                  |           | context deadline exc.  |
| https://10.30.0.13:2379  |                  |           | context deadline exc.  |
+--------------------------+------------------+-----------+------------------------+

$ ssh cp-1 'curl -s http://127.0.0.1:2381/metrics | grep -E "^etcd_server_has_leader|^etcd_server_leader_changes_seen_total"'

etcd_server_has_leader 0
etcd_server_leader_changes_seen_total 4418

$ ssh cp-1 'crictl logs --tail 20 $(crictl ps -a --name etcd -q | head -1)'

3a1f0b7c9d2e4a55 is starting a new election at term 4418
3a1f0b7c9d2e4a55 became candidate at term 4419
3a1f0b7c9d2e4a55 received MsgVoteResp from 3a1f0b7c9d2e4a55 at term 4419
3a1f0b7c9d2e4a55 [logterm: 12, index: 90418122] sent MsgVote request to 7c4e2d8a1b6f0933
3a1f0b7c9d2e4a55 [logterm: 12, index: 90418122] sent MsgVote request to c9b5e3f7a0d81244

$ kubectl describe node worker-07 | grep -A3 'Conditions:'

Conditions:
Type    Status  LastHeartbeatTime                 Reason
----    ------  -----------------                 ------
Ready   True    Tue, 14:07:12 +0000               KubeletReady

$ kubectl get nodes -l node-role.kubernetes.io/control-plane -o custom-columns='NODE:.metadata.name,ZONE:.metadata.labels.topology\.kubernetes\.io/zone'

NODE   ZONE
cp-1   dc-a
cp-2   dc-b
cp-3   dc-c

$ govc vm.info -json cp-1 cp-2 cp-3 | jq -r '.virtualMachines[] | [.name, .runtime.host.value] | @tsv'

cp-1   host-2201
cp-2   host-2214
cp-3   host-2214

$ ls -l /backup/etcd/ | tail -2 ; etcdutl snapshot status /backup/etcd/snapshot-1400.db -w table

-rw------- 1 root root 402653184 Aug 18 13:00 snapshot-1300.db
-rw------- 1 root root 403701760 Aug 18 14:00 snapshot-1400.db
+----------+----------+------------+------------+
|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| 8f2a19c4 | 90417806 |     241883 |     385 MB |
+----------+----------+------------+------------+

Work the evidence before reading on

The cluster is refusing writes and every one of its own reports is accurate.

1. Three members exist and one is answering. What is the quorum of a three-member cluster, and what does a lone member do when it campaigns?
2. kubectl get works and kubectl apply does not, and the node heartbeats are frozen rather than failing. What single property of the failure explains all three at once?
3. The zone labels say three domains and the hypervisor says two hosts. Which of those two statements was ever capable of being wrong, and which one was ever capable of being enforced?
4. The change record says one host and no impact. Given what the virtualisation team could see, was it wrong?

Before continuing: the members are powered off on a host that is rebooting, not destroyed. What does that fact change about the correct next action, and what is the cost of getting that classification wrong in each direction?

Root cause

1. Quorum counts failure domains, not members

A three-member cluster has a quorum of two and tolerates one loss. That arithmetic is worth something only if no single event can remove two members.

cp-2 and cp-3 were both guests on esx-14. Rebooting that host is one event and it removed two members, leaving cp-1 alone against a quorum of two. What followed is etcd working as designed: below quorum it refuses to commit rather than risk two halves diverging, and it refuses for as long as the condition lasts. There is no partial mode and no degraded write path.

In practice this was a two-domain cluster - tolerating zero losses of the domain holding two members - for eleven months.

2. Why the symptoms look like four different problems

Every symptom falls out of “writes fail, reads mostly answer”.

kubectl get works because a read does not need a commit; kubectl apply fails because it does. Running workloads are untouched because their Pods are already scheduled, their kube-proxy rules already programmed and their DNS records already in place - the data plane does not consult etcd to keep serving. The rollout is stuck at 4 of 12 because the Deployment controller cannot record its next decision, and the HPA has stopped for the same reason.

The frozen heartbeats are the most instructive of the four. A node condition is not failing; it is unchanged, because the kubelet’s update cannot be written. The cluster is reporting the last thing it managed to write down, which is the shape every observability signal takes when the store behind it stops accepting writes. A dashboard that has gone still and a dashboard that is green look identical.

3. The label was a claim and never a control

The topology.kubernetes.io/zone labels were typed in at install time to record the intended placement. Nothing derived them from anything, Kubernetes never checks them against reality, and the virtualisation platform has never read them.

So when a routine host evacuation moved cp-3 onto esx-14 eleven months ago, the platform did what it was asked, the labels went on saying dc-b and dc-c, and every design review, runbook and dashboard downstream of them kept describing a cluster that had stopped existing.

The change record was assessed honestly. One host, guests HA-protected, restart elsewhere within minutes is true of every guest on esx-14, including these two. It is simply insufficient when two of the guests are two thirds of a consensus cluster, and nothing in the change process could have noticed.

Resolution

1. Classify the loss before doing anything. Members powered off on a host that is rebooting are a transient loss; members whose disks are gone are a permanent one. The hypervisor console shows esx-14 posting through firmware, so this is transient and the correct action is to wait.
2. Turn the wait into an explicit hold with an owner and a deadline. The incident commander owns it; the deadline is the vendor stated firmware reboot time plus a margin, 14:45 here. Announce what happens at the deadline rather than deciding it there: restore the 14:00 snapshot onto a rebuilt three-member cluster with etcdutl snapshot restore, losing everything written after 14:00.
3. Copy the 14:00 snapshot somewhere that is neither esx-14 nor a control-plane host, and verify it again at the destination with etcdutl snapshot status. This is free, reversible, and it is the entire fallback plan.
4. Stop automation that retries writes in a tight loop. A GitOps controller hammering a cluster below quorum adds load to the one member that has to win an election the moment its peers return.
5. Do not add --force-new-cluster, and do not restart cp-1. One member cannot elect itself leader in a three-member cluster, so a restart costs the warm state and buys nothing.
6. Watch for the members returning rather than for the API server recovering. etcdctl endpoint status against all three is the signal; the API server following is a consequence.
7. When quorum returns, let the backlog drain before touching the stuck rollout. The Deployment controller resumes on its own, and a manual nudge during catch-up is how a recovered cluster acquires a second incident.
8. Before closing, confirm the hypervisor has not put cp-2 and cp-3 back onto one host, and raise the placement fix as a change rather than leaving it as a finding.

Verification

1. A leader exists and the members agree: etcdctl endpoint status -w table against all three shows exactly one IS LEADER true, with raft term and applied index matching rather than one trailing.
2. etcd_server_has_leader is 1 on all three members, read from each member rather than from one.
3. The cluster identity is unchanged. All three members report the cluster ID they had before 14:07. A new cluster ID means somebody forced, and that is a different incident with a different runbook.
4. A write round-trips end to end: create a scratch ConfigMap, read it back, delete it. Everything above can be true while the API server is still unhappy.
5. The frozen symptoms are moving: node LastHeartbeatTime advances on every node, and the rollout stuck at 4 of 12 completes on its own.
6. A fresh snapshot succeeds and etcdutl snapshot status reports a hash and revision. A successful snapshot is also evidence that the write path is genuinely healthy.
7. The placement is fixed and enforced, checked from the hypervisor inventory rather than from the Kubernetes labels - which is the check that was never run in eleven months.

Prevention

• Enforce the failure domain at the layer that can move the guests. A hypervisor anti-affinity rule or a placement group is a control. A topology.kubernetes.io/zone label is a string that Kubernetes will never verify and the virtualisation platform will never read.
• Where the label is all you have, audit it. Reconcile control-plane placement against the hypervisor inventory on a schedule and alert on drift, so that eleven months cannot pass unnoticed.
• Alert on quorum margin, not only on member health. A three-member cluster with one member down is one event away from this page and looks perfect from every application dashboard:

min(etcd_server_has_leader) == 0

count(up{job="etcd"} == 1) <= floor(count(up{job="etcd"}) / 2)

• Get the Kubernetes control plane into the virtualisation team’s change-impact model. “Single host, guests are HA-protected” is a true statement that evaluates the number of guests rather than what they are.
• Write the transient-versus-permanent decision into the runbook in advance, with the deadline and the stated data-loss cost of the fallback. It is not a decision anyone should make for the first time at 14:10 with a pipeline failing.
• Run a restore drill quarterly on a separate host. The deadline in that runbook is an honest number only if somebody has measured how long a restore actually takes here.