Three vs five members — design choices, write latency, fault tolerance

The choice of 3 versus 5 members is one of the most operationally significant decisions in standing up an etcd cluster. It determines fault tolerance, write latency, the cost of replacing a member, and the topology of the control plane. This lesson walks the trade-offs, the operating points, and the questions that drive the choice.

Two sizes, two design philosophies

flowchart LR
    subgraph Three[3-member etcd]
        L3[L]
        F3_1[F1]
        F3_2[F2]
    end
    subgraph Five[5-member etcd]
        L5[L]
        F5_1[F1]
        F5_2[F2]
        F5_3[F3]
        F5_4[F4]
    end

• 3 members — quorum 2; tolerates 1 failure; one round trip per write (leader + 1 ack).
• 5 members — quorum 3; tolerates 2 failures; one extra round trip per write (leader + 2 acks).

The fault-tolerance axis

The 5-member gains the ability to absorb a second simultaneous failure. Concrete scenarios where this matters:

• Lose a host (planned reboot crashes and won’t return), and at the same time one AZ partitions. With 3 members, this is quorum loss. With 5 members, the cluster continues.
• Lose a member to failed disk, and at the same time a network upgrade takes another member down briefly. With 3 members, this is quorum loss. With 5 members, it is degraded but serving.

The latency axis

Each additional member adds one peer round trip per write in the steady state. The cost:

In a single-AZ cluster with peer RTT of 0.5-1 ms, the extra round trip is sub-millisecond. In a multi-AZ cluster, peer RTT is 1-5 ms; the extra round trip is 2-5 ms.

$ etcdctl --endpoints=https://10.0.1.10:2379 --cacert=... --cert=... --key=... endpoint status --write-out=json | jq '.[] | {Endpoint, "raft applied index": .Status.raftAppliedIndex, "raft index": .Status.raftIndex, "db size": .Status.dbSize}'

[
{
  "Endpoint": "https://10.0.1.10:2379",
  "raft applied index": 41289312,
  "raft index": 41289312,
  "db size": 1500000000
}
]

The member-replacement axis

Replacing an etcd member involves:

1. Drain the member (remove it from the cluster’s membership: etcdctl member remove).
2. Build a fresh member (data dir, certificates, --initial-cluster).
3. Join the new member.

The cost: a few minutes of operational time per replacement. The frequency: whatever your rate of host replacement is.

Both are recoverable during member replacement; the 5- member has more headroom (3 healthy of 5 = quorum, even if one more host fails during rebuild).

The cost axis

5 members require:

• 5 dedicated hosts (stacked topology) or 5 dedicated etcd hosts (external topology).
• 5 sets of certificates (peer.crt, peer.key, server.crt, server.key).
• 5 members to operate on every kubeadm upgrade or init.
• 5 volumes for /var/lib/etcd.

The marginal cost over 3: 2 hosts, 2 sets of certificates, 2 maintenance units per change. In a cloud environment this is typically $100-200/month per member (small NVMe-backed VM with high-IOPS).

Failure-domain alignment

The choice of 3 or 5 should align with the failure-domain topology:

flowchart LR
    subgraph AZa[AZ-a]
        A1[m1]
        A2[m2]
    end
    subgraph AZb[AZ-b]
        B1[m3]
        B2[m4]
    end
    subgraph AZc[AZ-c]
        C1[m5]
    end
    AZa -.->|partition| AZb
    AZa -.->|partition| AZc

A 5-member cluster across three AZs (2/2/1) gives fault tolerance to:

• One host loss (any AZ).
• One AZ loss where the lost AZ is the single-member AZ (cluster loses one member; quorum still met).
• One AZ loss where the lost AZ is the two-member AZ (cluster loses two members; quorum still met for the remaining three members in the two surviving AZs).

A 5-member cluster across two AZs (3/2) is fragile to an AZ loss. A 5-member cluster across two AZs (2/2/1) is better but concentrates 1/5 of the cluster’s quorum weight in a single AZ.

When 7 members

Rarely:

• Strict 3-AZ spread with 3 members per AZ — but that is approximately 3+3+1, not even, and you still have the 1-AZ-quorum problem.
• Wider quorum tolerance than 5 — tolerating 3 failures. Tolerates an AZ with no members; tolerates a network partition with only 4 healthy. Most operators choose other mitigations (more aggressive maintenance, external etcd).

In practice, 7 is chosen by operators who want a 2-AZ-loss tolerance without external etcd, and who have budget for the extra hosts.

When external etcd

External etcd is a cluster of dedicated hosts running only etcd (no API server, scheduler, or controller manager). External etcd is chosen when:

• The control-plane size has grown such that co-locating etcd and the API server is too much for the hosts.
• The team requires separate operational ownership of etcd (often a database team).
• The cluster needs long-distance replication that kubeadm-stacked doesn’t address.

External etcd still runs 3 or 5 members with the same quorum arithmetic. The choice of size is the same.

The decision tree

flowchart LR
    Q1{Need to absorb a 2-host simultaneous failure?} -->|Yes| FIVE[5-member]
    Q1 -->|No| Q2{Multi-AZ?}
    Q2 -->|Yes| FIVE
    Q2 -->|No, single AZ| Q3{Budget and operational capacity for 5 hosts?}
    Q3 -->|Yes| FIVE
    Q3 -->|No| THREE[3-member]

The decision is rarely binary: many teams choose 5 for single-AZ clusters because the marginal cost is small and the headroom is genuine.

Migration between sizes

A 3-to-5 migration is done with etcdctl member add on the running 3-member cluster:

# From a healthy member, add a new member at name cp-4
etcdctl member add cp-4 \
  --peer-urls=https://10.0.1.13:2380

The cluster now has 4 members. Repeat to add cp-5. Now the cluster is 5 members. Quorum shifts from 2 to 3 as the 5th member joins and the cluster promotes it to follower.

A 5-to-3 reduction is possible but complex: remove members, and the cluster’s quorum size lags behind until the next cluster reconfiguration. Operators rarely do this; build a new cluster at the desired size and migrate workloads.

The TL;DR

• 3 is the default. Costs 3 hosts; tolerates 1 failure; one peer round trip per write.
• 5 is the production target. Costs 5 hosts; tolerates 2 failures; two peer round trips per write; absorbs an AZ loss or a host-plus-AZ simultaneous failure.
• 7 is rare. Larger cost with limited additional value.
• External is a topology choice, not a size choice. External etcd still runs 3 or 5 members.

Quiz

Production discipline

• 3 is the floor. No production cluster should run with 1 or 2 members.
• 5 is the target. The marginal cost over 3 is small and the fault-tolerance gain is genuine.
• Align member count with failure-domain count. Members divided across failure domains such that losing one failure domain leaves at least quorum of members.
• Don’t choose 7 unless you can articulate why. The default for production is 5; 7 is for the rare case that warrants the cost.
• Document the choice. The cluster’s fault tolerance is a function of member count and spread; the runbook should state both.

3 vs 5 is a design choice made at cluster build; document it, defend it, evolve it deliberately.