Failure domain placement — AZs, racks, hosts, and cost of spread

etcd’s fault tolerance is bounded by the failure domain that loses the most members in the worst case. If two members share a switch and the switch fails, two members are lost at once; if those two members are in a 3-member quorum of 2, the cluster is broken. Failure-domain placement is the discipline that aligns the cluster’s hardware topology with the fault tolerance the operator needs.

Failure domains in the order of frequency

flowchart TB
    HOST[host] --> RACK[rack] --> AZ[availability zone] --> REGION[region]
    HOST -.->|frequent| FRQ[common failure]
    RACK -.->|moderate| MOD[domain loss]
    AZ -.->|rare| RARE[AZ outages happen]
    REGION -.->|very rare| VRARE[multi-AZ regions are designed for AZ-loss, not region-loss]

The typical failure frequency:

etcd placement should cover host and rack routinely; AZ loss for production; region loss for the most stringent.

The 2/2/1 pattern

The most common production placement:

flowchart LR
    subgraph AZ1[AZ-a]
        M1
        M2
    end
    subgraph AZ2[AZ-b]
        M3
        M4
    end
    subgraph AZ3[AZ-c]
        M5
    end

The 2/2/1 placement is robust against any single AZ loss.

The 3/2 problem

The 5-member cluster spread 3/2 across two AZs:

The 3/2 placement is fragile to the loss of the AZ with 3 members. This is the common “we have 5 members but two AZs” failure.

The fix is to make the placement symmetric: 2/2/1 (3 AZs) or external etcd.

Host-level placement within an AZ

Within an AZ, members should be spread across:

• Hosts: each member on a different physical or virtual host.
• Racks: not always possible in a small AZ, but preferred.

A 3-member cluster in one AZ:

flowchart LR
    R1[Rack 1] --> M1
    R1 --> M2
    R2[Rack 2] --> M3

2 members in one rack, 1 in another. The rack with 1 member is the failure-domain-weak point within the AZ; the rack with 2 is the same risk as 2/1 spread. In a single AZ without multiple racks, host placement is the only dimension.

External etcd — failure-domain separation

External etcd moves the failure-domain choice for etcd out of the control-plane hosts:

flowchart LR
    subgraph ControlPlane[Control plane hosts]
        AS1[API server cp-1]
        AS2[API server cp-2]
        AS3[API server cp-3]
    end
    subgraph ExternalEtcd[External etcd hosts]
        E1[etcd member 1]
        E2[etcd member 2]
        E3[etcd member 3]
    end
    AS1 -.-> E1
    AS2 -.-> E2
    AS3 -.-> E3

External etcd places the etcd failure domain on dedicated hosts. The control-plane hosts can fail without touching etcd; etcd can fail without taking the control plane off the air (the API servers will error writes until etcd recovers).

flowchart LR
    AS -->|write attempt| E
    E -->|healthy| OK1[commit]
    E -->|down| FAIL[API server timeout]

The failure-domain separation is the reason this topology exists. The cost is two sets of hosts (3+3 = 6).

Cloud-specific AZ constraints

Cloud AZs have different failure characteristics:

The on-prem operator should document what their “AZ” means: is it two rooms? Two power feeds? Two switches?

Single AZ, single rack — the home-lab case

A home-lab or staging cluster typically has one host per “failure domain”. A 3-member cluster on one physical host three VMs is not placing etcd in multiple failure domains — all three etcd members share the host’s power and storage. The cluster is fault tolerant to member crash but not to host crash.

A 3-member cluster on three physical hosts in one rack places failure domain at the host level. This is the typical staging setup; it is not production-grade.

Designing the placement

For a new cluster, the placement decision:

1. Locate the failure domains. What are the independent failure domains in your environment?
2. Pick a size that aligns with their count. A cluster across 3 AZs wants 5 members (2/2/1); a cluster across 2 AZs wants 5 members but with the constraint that losing the AZ with more members is unsafe.
3. Place host-level within the AZs. Each member on a different host.
4. Place rack-level where possible. Distribute across rack PDUs and switches.
5. Document the placement. The runbook should state which member is in which AZ / rack / host.

Common anti-patterns

Quiz

Production discipline

• Spread at the failure-domain count. Members across as many independent failure domains as the deployment can sustain, with the cluster quorum surviving the loss of any one domain.
• Document the placement. Members to AZs to hosts to racks. The runbook must say this in one place.
• Re-validate spread on rebuilds. A member replaced on a different host is a placement change. Update the document.
• Plan migrations when topology changes. Changing the AZ layout requires a sequenced member add/remove operation; do not skip steps.

Failure-domain placement is the most common source of etcd outages that look like software bugs but are actually placement bugs.