Failure domain design — AZ placement and topology

The control plane’s failure tolerance depends on the spread of its components across failure domains. Two control-plane hosts in the same rack share that rack’s switch and power. Three etcd members in the same AZ are vulnerable to AZ loss. This lesson walks the failure domain placement decisions that harden the control plane against correlated failure.

Failure domains in order of frequency

flowchart TB
    H[Host] --> R[Rack] --> AZ[Availability zone] --> REG[Region]
    H -.->|most common| FRQ1[host failure]
    R -.->|moderately common| FRQ2[rack failure]
    AZ -.->|rare| FRQ3[AZ outage]
    REG -.->|very rare| FRQ4[regional outage]

The frequency of failure determines how aggressively the design must spread:

• Host failures are routine maintenance events.
• Rack failures are uncommon but real (switch, PDU).
• AZ failures are rare in cloud; rarer in on-prem.
• Region failures are very rare; multi-region is aspirational.

The etcd placement

A 5-member etcd cluster should be 5/0/0, 3/2/0, or 2/2/1 across 1-3 AZs:

flowchart LR
    subgraph AZa[AZ-a]
        M1
        M2
    end
    subgraph AZb[AZ-b]
        M3
        M4
    end
    subgraph AZc[AZ-c]
        M5
    end
    AZa -.->|loss| M1
    AZa -.->|loss| M2
    AZb -.->|loss| M3
    AZb -.->|loss| M4
    AZc -.->|loss| M5

A 2/2/1 spread gives AZ-loss tolerance for any one AZ; any two-AZ loss loses quorum.

The control-plane placement

The control-plane hosts (where API server, scheduler, controller manager run) should also be spread:

flowchart LR
    subgraph AZa[AZ-a]
        cp1[cp-1: API server + etcd]
    end
    subgraph AZb[AZ-b]
        cp2[cp-2: API server + etcd]
    end
    subgraph AZc[AZ-c]
        cp3[cp-3: API server + etcd]
    end

For stacked topology: each control-plane host is in a different AZ. AZ loss takes one control-plane host; the cluster’s two remaining members have quorum.

For external topology:

flowchart LR
    subgraph CP-AZs[Control plane AZs]
        cp1[cp-1]
        cp2[cp-2]
    end
    subgraph ETCD-AZs[Etcd AZs]
        e1[etcd-1]
        e2[etcd-2]
        e3[etcd-3]
    end
    cp1 -.- e1
    cp2 -.- e2
    cp3 -.-> e3

External topology allows control-plane hosts to live where they are easiest to operate (e.g., edge or datacenter); etcd is on dedicated infrastructure.

The LB placement

The LB hosts must be in different failure domains than each other and may share AZ with the API servers:

flowchart LR
    subgraph AZa[AZ-a]
        LB1[LB host 1]
        cp1[API server cp-1]
    end
    subgraph AZb[AZ-b]
        LB2[LB host 2]
        cp2[API server cp-2]
    end
    AZa -.->|loss| LB1
    AZa -.->|loss| cp1
    AZb -.->|loss| LB2
    AZb -.->|loss| cp2

LB host 1 carries the VIP normally. On AZ-a loss:

• LB1 lost; VIP moves to LB2 (in AZ-b).
• cp-1 lost; cluster continues on cp-2 and cp-3.
• The VIP on LB2 routes to cp-2 and cp-3.
• The cluster survives.

On both-AZ loss: cluster unreachable (correlated AZ failures are rare).

$ kubectl get nodes -o wide --label-columns=topology.kubernetes.io/zone,topology.kubernetes.io/region

...

The rack-level placement

On-prem clusters often have racks as failure domains:

flowchart LR
    subgraph Rack1[Rack 1]
        cp1[cp-1]
        cp2[cp-2]
    end
    subgraph Rack2[Rack 2]
        cp3[cp-3]
    end

A 3-host cluster in 2 racks: 2 in rack 1, 1 in rack 2. The single-rack failure loses 1 host (rack 2) or 2 hosts (rack 1). With quorum of 2 and tolerance of 1, the rack loss depends on which rack fails:

• Rack 1 loss: cp-3 remains; quorum lost.
• Rack 2 loss: cp-1, cp-2 remain; quorum preserved.

Rack placement matters; spread to 3 racks for full rack tolerance.

The network domain

The cluster’s network path is a failure domain:

• Network switch / router in the cluster’s path.
• VLAN / subnet shared with other workloads.
• ISP link for cloud-managed clusters.
• ToR switch in a rack.

The control plane’s network path should be isolated from noisy-neighbour workloads and redundant at the protocol level (multi-link LACP, dual switches).

The power domain

Power feeds:

• Single PDU per rack — losing that PDU loses the rack.
• Dual PDU per rack — losing one PDU is tolerated; the other powers the rack.
• UPS — short outage tolerated.
• Generator — long outage tolerated.

The control plane’s hosts should be on dual PDU feeds.

The correlated failure checklist

A failure-domain analysis includes:

• Hosts: on different physical hosts.
• Racks: on different racks (if 3+ racks available).
• Switches: on different switches (dual ToR).
• PDUs: on different PDUs (dual feed).
• AZs: on different AZs (cloud clusters).
• Backups: off-cluster, ideally off-AZ.
• DNS: master DNS is in one AZ; backup in another.

A correlated failure loses the cluster if multiple domains collapse simultaneously. Production design spreads widely.

The “good enough” answer

For most production clusters:

• Stacked topology on 3 hosts in 3 AZs (cloud) or 3 racks (on-prem).
• 5-member etcd where the cluster size justifies the cost.
• LB on 2 hosts in 2 different failure domains.
• DNS in 2 AZs.

This setup tolerates host failure, AZ loss (most), rack failure (on-prem most), and recovers from any single one.

The discipline

• Document the failure domain map. Each member’s host, rack, AZ, PDU, switch.
• Test the failure domain. Drill: kill an entire AZ and verify the cluster remains.
• Avoid single-domain leader. Any single component that runs on one host without HA is a correlated risk.
• Spread backups. Snapshots stored off-cluster, ideally off-AZ.

Quiz

Production discipline

• Spread across AZs. Stacked topology’s 3 hosts in 3 AZs gives AZ-loss tolerance.
• Spread across racks / PDUs. On-prem clusters benefit from rack-level isolation.
• Avoid single-domain leaders. Any single-host single- process component is a correlated risk.
• Document the spread. The runbook should name the AZ / rack / PDU for each host.
• Test the loss. A chaos test that takes an AZ offline validates the design.

Failure domain placement is the foundation of cluster HA. Operating it well is documented and tested.