Quorum loss triage — the cluster's brain failure

Lose two of three etcd members and the cluster stops accepting writes. Running Pods keep serving traffic and reads still work, but every kubectl apply fails and no controller can record a decision. What has to be settled first is whether the loss is transient — members restarting, a brief partition — or permanent, because waiting costs minutes while forcing a new cluster or restoring a snapshot costs every write since the last backup. This lesson covers reading the member list and endpoint health to tell the two apart before touching anything.

The etcd quorum

The etcd’s quorum is the cluster’s brain. The quorum is the minimum number of members that must be available for the cluster to function. For a 3-member cluster, the quorum is 2. For a 5-member cluster, the quorum is 3.

flowchart TD
    A[3-member etcd] --> B{Quorum of 2}
    B -->|Yes| C[Healthy]
    B -->|No| D[Brain failure]
    D --> E[API server cannot write]

The quorum is the etcd’s health.

The diagnostic

The canonical diagnostic:

# Substitute your own values before running:
ETCD_NODE=cp-01
PEER_ETCD_NODE=cp-02

# 1. Check the etcd's members
ETCDCTL_API=3 etcdctl member list

# 2. Check the etcd's endpoint status
ETCDCTL_API=3 etcdctl endpoint status

# 3. Check the etcd's health
ETCDCTL_API=3 etcdctl endpoint health

# 4. Check the etcd's logs
ssh "$ETCD_NODE" "crictl logs --tail 200 \$(crictl ps -a --name etcd -q | head -1)"

# 5. Check the network
ping -c 4 "$PEER_ETCD_NODE"

The diagnostic is the members, the endpoint status, the health, and the logs.

Quorum loss

A quorum loss is the etcd’s signal that the cluster is non-functional. The causes:

• Two members failed. Two of three members are down.
• Network partition. The members cannot reach each other.
• Split brain. The members are partitioned into two groups, each with a partial quorum.

flowchart TD
    A[Quorum loss] --> B{Two members failed?}
    B -->|Yes| C[Recover the failed members]
    B -->|No| D{Network partition?}
    D -->|Yes| E[Fix the network]
    D---|No| F{Split brain?}
    F -->|Yes| G[Force a new cluster]
    F -->|No| H[Unknown]

The remediation

The remediation depends on the cause:

# Substitute your own value before running:
FAILED_ETCD_NODE=cp-02

# Option 1: Recover the failed members.
# On a kubeadm cluster etcd is a static Pod, not a systemd
# service, so restart it by moving its manifest aside and back.
ssh "$FAILED_ETCD_NODE" "mv /etc/kubernetes/manifests/etcd.yaml /tmp/ && sleep 25 && mv /tmp/etcd.yaml /etc/kubernetes/manifests/"

# Option 2: Force a new single-member cluster (last resort).
# --force-new-cluster is an etcd server flag, not an etcdctl
# one: add it to the command line in
# /etc/kubernetes/manifests/etcd.yaml on the surviving member,
# then remove it again once the member is up. It panics if any
# other member from the old cluster is still alive.

# Option 3: Restore from a snapshot
etcdutl snapshot restore /var/backups/etcd/snapshot.db

The remediation is the etcd recovery.

The quorum math

For a 3-member cluster:

• 3 healthy: quorum (2 + 1) is met. Healthy.
• 2 healthy: quorum (2) is met. Healthy.
• 1 healthy: quorum is not met. Brain failure.

For a 5-member cluster:

• 5 healthy: quorum (3 + 2) is met. Healthy.
• 4 healthy: quorum (3) is met. Healthy.
• 3 healthy: quorum (3) is met. Healthy.
• 2 healthy: quorum is not met. Brain failure.

The quorum is the cluster’s threshold.

The transient loss

A transient loss is temporary. The members are restarting; the network is partitioned briefly. The remediation is to wait.

# Substitute your own value before running:
ETCD_NODE=cp-01

# Check the logs
ssh "$ETCD_NODE" "crictl logs --tail 200 \$(crictl ps -a --name etcd -q | head -1)"

# Wait for the members to recover

The transient loss is the cluster’s hiccup.

The permanent loss

A permanent loss is not recoverable. The members are dead. The remediation is to force a new cluster (with data loss) or restore from a snapshot.

# Force a new cluster (data loss)
ETCDCTL_API=3 etcdctl --force-new-cluster member add ...

# Restore from a snapshot
etcdutl snapshot restore /var/backups/etcd/snapshot.db

The permanent loss is the cluster’s worst day.

Production discipline

A quorum loss is the cluster’s hypothesis. The discipline is to walk the 11-step methodology applied to the etcd, identify the cause, apply the remediation. The etcd is the cluster’s state; the remediation is the etcd recovery.

• Check the etcd’s members. The members are the etcd’s nodes.
• Check the etcd’s health. The health is the etcd’s state.
• Restart the failed members. The restart is the etcd’s recovery.

Quiz