etcd member failure

Reported symptoms

Four tickets are open against prod-eu, assigned to four different teams, and all four were filed nine days ago.

Delivery has one about flaky CI: the GitOps reconciler fails roughly one apply in five with etcdserver: request timed out, and the retry always works, so the agreed workaround is to retry. Network has one about the control-plane load balancer, which keeps marking a backend down and then healthy again several times an hour; nothing has changed on the load balancer. Platform has one about the nightly etcd snapshot, failing every night with context deadline exceeded, triaged low because a snapshot file already exists on disk. Security has one about a certificate pre-check that could not reach etcd from cp-2; it was rerun from cp-1, succeeded, and was closed as transient.

Meanwhile the cluster looks fine. All three control-plane nodes are Ready. All three etcd-cp-* static Pods are Running. kubectl works. Nothing has alerted, because this cluster has no etcd alert at all - the assumption when it was built was that the monitoring stack covered it.

Evidence provided

$ kubectl get pods -n kube-system -l component=etcd -o wide

NAME        READY   STATUS    RESTARTS         AGE   NODE
etcd-cp-1   1/1     Running   0                63d   cp-1
etcd-cp-2   1/1     Running   412 (2m14s ago)  63d   cp-2
etcd-cp-3   1/1     Running   0                63d   cp-3

$ kubectl -n kube-system exec etcd-cp-1 -- etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key member list -w table

+------------------+---------+------+--------------------------+
|        ID        | STATUS  | NAME |        PEER ADDRS        |
+------------------+---------+------+--------------------------+
| 3a1f0b7c9d2e4a55 | started | cp-1 | https://10.20.0.11:2380  |
| 7c4e2d8a1b6f0933 | started | cp-2 | https://10.20.0.12:2380  |
| c9b5e3f7a0d81244 | started | cp-3 | https://10.20.0.13:2380  |
+------------------+---------+------+--------------------------+

$ kubectl -n kube-system exec etcd-cp-1 -- etcdctl --cacert=... --cert=... --key=... --endpoints=https://10.20.0.11:2379,https://10.20.0.12:2379,https://10.20.0.13:2379 endpoint health --cluster -w table

+--------------------------+--------+---------+---------------------------+
|         ENDPOINT         | HEALTH |  TOOK   |           ERROR           |
+--------------------------+--------+---------+---------------------------+
| https://10.20.0.11:2379  | true   | 4.113ms |                           |
| https://10.20.0.13:2379  | true   | 5.902ms |                           |
| https://10.20.0.12:2379  | false  | 5.0002s | context deadline exceeded |
+--------------------------+--------+---------+---------------------------+

$ ssh cp-2 'crictl logs --tail 40 $(crictl ps -a --name etcd -q | head -1)'

slow fsync  took="0.871s"  expected-duration="1s"
failed to send out heartbeat on time  heartbeat-interval="100ms"
lost leader  local-member-id="7c4e2d8a1b6f0933"
raft: 7c4e2d8a1b6f0933 lost leader c9b5e3f7a0d81244 at term 4417

$ curl -s 'http://prometheus.prod-eu.svc:9090/api/v1/query?query=histogram_quantile(0.99,sum(rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m]))by(le,instance))'

cp-1   0.0041
cp-3   0.0053
cp-2   0.8907

$ ssh cp-2 'sudo storcli /c0 show | grep -iE "cache|bbu"'

BBU Status = Failed
Current Cache Policy = WriteThrough, ReadAhead, Direct

$ ssh cp-1 'grep etcd-servers /etc/kubernetes/manifests/kube-apiserver.yaml'; kubectl -n ops get cronjob etcd-snapshot -o jsonpath='{.spec.jobTemplate.spec.template.spec.nodeName}'

    - --etcd-servers=https://127.0.0.1:2379
cp-2

Work the evidence before reading on

Four teams have four tickets and there is one fault.

1. member list says started and endpoint health says false for the same member. What is each command actually asking, and which one is a claim about health?
2. Every kube-apiserver here points at 127.0.0.1:2379. What happens to a request the load balancer sends to cp-2 while the member on cp-2 is restarting?
3. WAL fsync p99 on cp-2 is 890 ms, and the etcd static Pod has a liveness probe. What does the kubelet do to a container whose liveness probe times out, and what does that do to the fsync latency?
4. Three members, two healthy. How much failure tolerance is left, and how long has that been true?

Before continuing: name the single hardware fact that produces the flaky applies, the flapping load-balancer backend, the failing backup job, and the unreachable certificate pre-check, without needing a second fault to explain any of them.

Root cause

1. A dead write cache is an etcd outage with a delay

The RAID controller on cp-2 lost its battery-backed write cache and fell back to write-through. Every write on that host now waits for the device instead of being acknowledged from a protected cache, and fsync latency went from about 4 ms to about 890 ms.

For most workloads that is a performance ticket. For etcd it is fatal, because the WAL fsync sits on the commit path: nothing is committed until it is durably persisted, so the member’s service rate collapses to the disk’s. It stops answering its own health endpoint inside the probe timeout, the kubelet does what a failing liveness probe means and kills the container, and the container restarts to replay its WAL onto the same slow disk. Four hundred and twelve times so far.

2. started is a membership fact, not a health fact

The most misleading line in the evidence is the reassuring one. etcdctl member list reports every member as started, including the one that has been unavailable for nine days.

started means the member exists in the membership table and completed its initial bootstrap. It is a statement about configuration, not liveness. The cluster never removes a member for failing to answer - removal is an operator action - so a member whose process is dead reads as started until somebody removes it. endpoint health is the command that asks a live question, and it is the one returning false with context deadline exceeded.

The same trap sits in kubectl get pods. STATUS: Running is true: the kubelet restarted the container two minutes ago and it is running now. The column carrying the information is RESTARTS, and it is the column that scrolls past.

3. kubeadm points each API server at its own member only

kubeadm writes --etcd-servers=https://127.0.0.1:2379 into the kube-apiserver manifest on every control-plane host. Each API server talks to the etcd member on its own machine and to no other.

That default has a consequence here. When the member on cp-2 is down, the API server on cp-2 cannot serve anything that touches storage: readiness fails, the load balancer drops the backend, the member restarts, readiness recovers, and the backend returns. The network team’s flapping ticket is a faithful report of a member restarting a few dozen times a day.

It is also the delivery team’s ticket. Requests arriving at cp-2 in the window between the member dying and readiness noticing fail with etcdserver: request timed out - roughly one in five, because there are three backends and the window is short. The retry lands elsewhere and works, which is exactly why it read as flaky CI.

4. The backup stopped on the same day

The snapshot CronJob is pinned to cp-2 by nodeName, so it has been failing since the disk did. The newest verified snapshot of this cluster is nine days old, and nobody escalated because the ticket said “job failed” rather than “cluster has no recent backup”.

Resolution

1. Take a snapshot from cp-1 or cp-3 and verify it before touching membership. etcdctl snapshot save is a live API call against a healthy endpoint; etcdutl snapshot status on the result is what makes it a backup rather than a file. How old that file is decides how much risk everything below carries.
2. Stop the churn, which is a separate action from fixing the member. Move kube-apiserver.yaml aside on cp-2 so the load balancer stops flapping, and etcd.yaml aside so the kubelet stops restarting a member that cannot succeed. The cluster is now cleanly two-of-three.
3. Decide whether to proceed or hold, and record the decision. Holding is legitimate here - the cluster serves and the noise is gone - but not indefinitely, because two-of-three has zero loss tolerance. Name the owner of the cp-2 hardware and the hour by which the disk is repaired or the member replaced.
4. Repair the hardware before rejoining anything. A member that rejoins onto the same write-through controller fails the same way, and it fails while the cluster has one fewer healthy member than it does now.
5. Remove the failed member: etcdctl member remove 7c4e2d8a1b6f0933 from cp-1. Membership becomes two, quorum becomes two, and the two healthy members satisfy it. Confirm with member list that the entry is gone before going further; a stale entry means the configuration change did not commit.
6. Rebuild cp-2: kubeadm reset on the host, clear /var/lib/etcd, and kubectl delete node cp-2 if it rejoins under the same name.
7. Rejoin: kubeadm init phase upload-certs --upload-certs on cp-1 to mint a certificate key, then kubeadm join with --control-plane on cp-2, which performs the member add and writes both manifests.
8. Return cp-2 to the load-balancer pool only after its API server reports ready on its own, and unpin the snapshot CronJob before closing. The backup gap is part of this incident, not a follow-up.

Verification

1. Membership is three and clean: etcdctl member list -w table shows three started members and no leftover entry for the removed ID.
2. Consensus is real: etcdctl endpoint status -w table --cluster shows exactly one member with IS LEADER true, and the raft term and applied index agree across all three rather than one trailing.
3. Health is live rather than tabular: etcdctl endpoint health --cluster returns true for all three endpoints with response times in the same order of magnitude.
4. The disk is fixed, which is the check that proves the diagnosis. etcd_disk_wal_fsync_duration_seconds p99 on the rebuilt member sits with cp-1 and cp-3 in single-digit milliseconds. If it is 890 ms again, the member was replaced and the fault was not.
5. The restart count on the rebuilt etcd Pod is still zero an hour later - the symptom that was visible the whole time and that nobody read.
6. The load-balancer backend has stopped flapping and a full GitOps reconcile completes with no etcdserver: request timed out.
7. A snapshot succeeds from each of the three hosts in turn and etcdutl snapshot status reports a hash and a revision for each.
8. The new alerts can fire. On a staging cluster, stop one member and confirm the member-down alert arrives. An alert that has never fired is a configuration file, not a control.

Prevention

• Alert on the three etcd facts this cluster could not see. Member count and leadership catch a member that has left; per-member fsync latency catches the disk before the member leaves:

histogram_quantile(0.99, sum(rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])) by (le, instance)) > 0.025

max_over_time(etcd_server_has_leader[5m]) == 0

• Alert on restart counts for control-plane static Pods. A container on its four hundred and twelfth restart is Running, and the word is worthless here.
• Alert on backup age, not on backup job exit status, and never pin the snapshot job to a single host. A backup produced only by the machine most likely to be the reason you need one is not a backup.
• Split the member-replacement runbook on failed versus healthy, because the safe order is opposite in the two cases, and write the quorum arithmetic beside each branch so the next reader can check it rather than trust it.
• Give “degraded but serving” a service level with a clock. Three members with one down is zero margin, and nothing in the cluster will tell you how long it has lasted.
• When four teams file four tickets on the same day, treat the date as the strongest evidence in the incident.