Quorum math — floor(N/2)+1, odd members, fault tolerance

etcd’s quorum is the single number that defines what “healthy” and “broken” mean for the cluster. Past quorum, no commit; under quorum, no recovery short of restoring from a snapshot. This lesson walks the arithmetic, the operating points for production cluster sizes, and the boundary between “degraded but serving” and “quorum lost, restore from snapshot”.

Quorum in one sentence

A commit is acknowledged when a majority of members have persisted it to their log. The majority size is floor(N/2) + 1 where N is the cluster size. A cluster loses quorum when fewer than that many members can participate in a write.

The quorum table

flowchart TB
    N["Cluster of N"] -->|Q| Qq["quorum = floor(N/2) + 1"]
    Qq -->|loss tolerance| L["loss = N - quorum"]

The single number every operator needs:

• Quorum is floor(N/2) + 1.
• Loss tolerance is N - quorum = floor(N/2).
• Odd sizes align loss tolerance with quorum cleanly; even sizes pay one extra member for the same fault tolerance.

Why odd numbers

A 4-member cluster has the same fault tolerance as a 3-member cluster (1 lost = quorum loss if 1 was already unreachable; well, 3 tolerates 1 loss, 4 also tolerates 1 loss). The 4-member costs an extra member for no resilience gain. A 5-member tolerates 2 losses; a 6-member tolerates 2 losses too. The 5 is the better ratio.

A counter-argument: odd numbers split evenly under a two-way partition, even numbers create a 3-3 or 4-4 split where neither side has quorum but could have had it with one fewer member on each side. In a balanced two-side partition, 5 members produce a 2-3 split — one side has quorum, the other does not; the cluster commits on the winning side. A 6-member cluster produces a 3-3 split — neither side has quorum.

Members down vs quorum lost

flowchart LR
    H[Healthy cluster N=3] -->|1 member down| D[Degraded, quorum kept]
    D -->|2nd member down| QL[Quorum lost, writes halt]
    QL -->|1 returns| D
    QL -->|nothing returns| STUCK[Cluster stuck; restore from snapshot]

“Members down” and “quorum lost” are distinct states:

• Members down, quorum kept: the remaining members hold an election; one wins; the cluster continues. This is degraded but serving.
• Quorum lost: no election can complete; no leader can be elected; no write can be replicated; the cluster’s API requests time out. This is broken and only snapshot restore can rescue it.

# Verify cluster member state
etcdctl member list --write-out=table

+------------------+---------+---------------+--------------------------+--------------------------+
|        ID        | STATUS  |     NAME      |       PEER ADDRS         |      CLIENT ADDRS        |
+------------------+---------+---------------+--------------------------+--------------------------+
| c5e9a1b2...     | started | cp-1          | https://10.0.1.10:2380  | https://10.0.1.10:2379  |
| d7a1c8f3...     | started | cp-2          | https://10.0.1.11:2380  | https://10.0.1.11:2379  |
| e8b4d2a5...     | unstarted | cp-3       | https://10.0.1.12:2380  | https://10.0.1.12:2379  |
+------------------+---------+---------------+--------------------------+--------------------------+

A started member is healthy in the cluster’s view; unstarted is unusual (often indicates a member whose data dir has been wiped but whose name still appears in the cluster membership list — see Part LXVII lesson 4 on member lifecycle).

Quorum arithmetic for the common sizes

Three members

• Quorum: 2.
• Loss tolerance: 1.
• Cost: minimum for production. Common in tightly-coupled control planes.

Five members

• Quorum: 3.
• Loss tolerance: 2.
• Cost: most common production target. Tolerates two simultaneous failures (one host failure, one AZ partition, for example).

Seven members

• Quorum: 4.
• Loss tolerance: 3.
• Cost: rarely chosen. Each added member pays the full per-write replication latency; the cluster slows linearly with member count.

Quorum lost — the boundary condition

Quorum is lost when the cluster’s healthy member count falls below quorum:

• 3 cluster: lost if 2 are down.
• 5 cluster: lost if 3 are down.
• 7 cluster: lost if 4 are down.

In each case, the leader (if still considered leader by its own view) cannot replicate to a majority. The cluster refuses writes. The API server’s writes time out. The cluster does not “auto-recover” — there is no quorum to hold an election.

sequenceDiagram
    autonumber
    participant A as Member A
    participant B as Member B
    participant C as Member C
    Note over A,C: 3-member, partitioned 2-1
    A->>B: AppendEntries (peer ok)
    A-->>A: ack count = 1 (insufficient quorum)
    A-->>C: cannot reach
    Note over A: leader stays leader by own view,<br/>but cannot commit
    A->>A: API server writes time out

The error pattern at the API server:

Internal error occurred: failed calling webhook "xxx":
Post "https://admission.example/...": context deadline exceeded

Or, directly from etcd:

etcdserver: request timed out

A working cluster shows these within seconds when quorum is lost. The mitigation is not “wait it out”; it is restore from the most recent snapshot.

Why “wait it out” never works

If quorum is lost because of a network partition, the recovery requires the network to heal. If quorum is lost because two members crashed simultaneously, the recovery requires restoring those members (which still takes hours, not minutes). If quorum is lost because data was corrupted on multiple members, the only recovery is restore from snapshot.

The operator’s mental model: quorum loss is a disaster-class event, not a performance regression.

Quorum in the production topology

Three practical patterns:

1. Stacked etcd on the control plane (kubeadm default). 3 members, all on the control-plane hosts. Quorum loss means 2 control-plane hosts are unreachable.
2. Stacked etcd with 5 members. 5 control-plane hosts, each with an etcd member. Quorum loss means 3 hosts are unreachable.
3. External etcd cluster. 3 dedicated etcd hosts, separate from the API server hosts. Quorum loss is a fault in the dedicated etcd infrastructure, not in the control plane.

In each case, the numbers apply identically; the answer is always floor(N/2)+1.

The frequently-made mistakes

$ etcdctl --endpoints=https://10.0.1.10:2379,... --cacert=... --cert=... --key=... endpoint status --write-out=table

+---------------------------+------------------+---------+---------+-----------+
|         ENDPOINT          |        ID        | IS LEADER | RAFT TERM |  RAFT INDEX |
+---------------------------+------------------+---------+---------+-----------+
| https://10.0.1.10:2379    | c5e9a1b2...      |   true   |    17    |  41289312 |
| https://10.0.1.11:2379    | d7a1c8f3...      |  false   |    17    |  41289312 |
| https://10.0.1.12:2379    | e8b4d2a5...      |  false   |    17    |  41289312 |
+---------------------------+------------------+---------+---------+-----------+

A consistent RAFT INDEX across members is the quick visible sanity check. Drift indicates one member has not applied the latest committed entries.

Quiz

Production discipline

• Use 3 or 5 members in production. Odd numbers, aligned with failure-domain count. Never 4 or 6.
• Document the quorum size in the runbook. Every operator who touches the cluster should be able to recite the quorum (3) and the loss tolerance (1) without counting.
• Alert when any member is unstarted. That state is meaningful and almost always indicates a problem; it does not appear in a healthy cluster.
• Treat quorum loss as disaster class. Recovery is restore from snapshot, not “fix the network and wait”.
• Plan AZ spread for failure-domain tolerance. Two members per AZ, one AZ with one member, gives 2-member AZ-loss tolerance only if the AZ with one member is the one that fails.

Quorum is the line. Past it, the cluster works. Across it, the cluster is broken. Operations that cross the line are not performance tasks; they are disaster recovery tasks.