Replace failed member — the cluster's recovery

A three-member etcd cluster that has lost one member still has quorum, but it has no margin left: the next failure stops every write and takes the API server with it. The order of the repair is what matters — remove the dead member before adding its replacement, because adding first takes the cluster to four members and raises quorum to three, which two healthy members cannot satisfy. This lesson covers identifying the failed member, the remove-then-add sequence, and the kubeadm path for a node that hosts both etcd and the API server.

The failed member

A failed member is an etcd member that is not participating in the cluster. The cluster continues to function as long as the quorum is met.

flowchart TD
    A[3-member etcd] --> B{All members healthy?}
    B -->|Yes| C[Healthy]
    B -->|No| D{Quorum met?}
    D -->|Yes| E[Replace the failed member]
    D -->|No| F[Quorum loss]

The failed member is the cluster’s degraded state.

The diagnostic

The canonical diagnostic:

# Substitute your own values before running:
FAILED_ETCD_NODE=cp-03.example.com

# 1. Check the etcd's members
ETCDCTL_API=3 etcdctl member list

# 2. Check the etcd's endpoint status
ETCDCTL_API=3 etcdctl endpoint status

# 3. Check the etcd's health
ETCDCTL_API=3 etcdctl endpoint health

# 4. Check the failed member's logs
ssh "$FAILED_ETCD_NODE" "journalctl -u etcd -n 200"

# 5. Check the cluster's state
kubectl get nodes

The diagnostic is the members, the endpoint status, the health, and the logs.

The replacement

The canonical replacement:

# Substitute your own values before running:
# MEMBER_ID is the failed member's hex id from `etcdctl member list` above.
MEMBER_ID=8e9e05c52164694d
NEW_NAME=cp-03
NEW_NODE=192.0.2.13
# Every member of the cluster, including the one being added:
INITIAL_CLUSTER=cp-01=https://192.0.2.11:2380,cp-02=https://192.0.2.12:2380,cp-03=https://192.0.2.13:2380
CLUSTER_TOKEN=etcd-cluster-1

# 1. Remove the failed member from the cluster
ETCDCTL_API=3 etcdctl member remove "$MEMBER_ID"

# 2. Add a new member
ETCDCTL_API=3 etcdctl member add "$NEW_NAME" \
  --peer-urls="https://$NEW_NODE:2380"

# 3. Start the new member
etcd --name "$NEW_NAME" \
  --initial-cluster "$INITIAL_CLUSTER" \
  --initial-cluster-token "$CLUSTER_TOKEN" \
  --initial-advertise-peer-urls "https://$NEW_NODE:2380" \
  --advertise-client-urls "https://$NEW_NODE:2379" \
  --data-dir /var/lib/etcd

# 4. Verify the cluster
ETCDCTL_API=3 etcdctl endpoint health

The replacement is the cluster’s recovery.

The kubeadm-managed replacement

The kubeadm-managed replacement is the alternative to the manual replacement:

# Substitute your own values before running:
FAILED_ETCD_NODE=cp-03.example.com
NEW_NODE=cp-04.example.com
LOAD_BALANCER=cp.example.com
# Token and CA cert hash from `kubeadm token create --print-join-command`,
# certificate key from `kubeadm init phase upload-certs --upload-certs`,
# both run on a surviving control-plane node:
JOIN_TOKEN=9vr2v5.3xk8p1q0mzn7wd4t
CA_CERT_HASH=sha256:a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f90
CERT_KEY=f8e7d6c5b4a39281706f5e4d3c2b1a09f8e7d6c5b4a39281706f5e4d3c2b1a09

# 1. Remove the failed member from the cluster
# Edit /etc/kubernetes/manifests/etcd.yaml on the remaining members

# 2. Use kubeadm to reset the failed member
ssh "$FAILED_ETCD_NODE" "kubeadm reset"

# 3. Re-join the new member
ssh "$NEW_NODE" "kubeadm join $LOAD_BALANCER:6443 --token $JOIN_TOKEN --discovery-token-ca-cert-hash $CA_CERT_HASH --control-plane --certificate-key $CERT_KEY"

The kubeadm-managed replacement is the cluster’s default.

The remediation

The remediation depends on the failure:

# Option 1: Manual replacement
ETCDCTL_API=3 etcdctl member remove
ETCDCTL_API=3 etcdctl member add

# Option 2: kubeadm-managed replacement
kubeadm reset
kubeadm join

The remediation is the member replacement.

Production discipline

A failed member is the cluster’s hypothesis. The discipline is to walk the 11-step methodology applied to the etcd, identify the cause, apply the remediation. The etcd is the cluster’s state; the remediation is the member replacement.

• Check the etcd’s members. The members are the etcd’s nodes.
• Check the etcd’s health. The health is the etcd’s state.
• Remove before you add. The failed member is removed first; adding first takes a three-member cluster to four and raises quorum to three, which two healthy members cannot satisfy.

Quiz