Failure-domain modelling

Why this matters in production

A “highly available” cluster with all three nodes in the same rack, fed from the same PDU, on the same network switch is not highly available — it has one failure domain. The moment that rack, PDU, or switch fails, the entire cluster goes with it.

Mental model

A failure domain is the set of components that fail together when a single underlying cause occurs. Common domains:

flowchart TB
  subgraph R1[Rack 1 - failure domain A]
    S1[pve-01] --- SW1[ToR switch 1]
  end
  subgraph R2[Rack 2 - failure domain B]
    S2[pve-02] --- SW2[ToR switch 2]
  end
  subgraph R3[Rack 3 - failure domain C]
    S3[pve-03] --- SW3[ToR switch 3]
  end
  SW1 --- NET[Network fabric]
  SW2 --- NET
  SW3 --- NET

Failure-domain analysis for a Proxmox cluster

For each component, list the failure domains it shares:

Designing against correlated failure

The goal is to make every critical path cross at least two independent failure domains. For the cluster quorum path (Corosync), this means:

• Two independent switches (or LACP across one switch’s two supervisor modules).
• Two independent paths between any pair of nodes.
• Distinct uplinks.

For storage (Ceph), this means:

• OSDs placed across failure domains (hosts, racks, rooms).
• CRUSH rules that keep replicas in different domains.
• Network paths that use distinct switches from the cluster network.

For backups, this means:

• PBS in a separate rack or site from the primary cluster.
• PBS datastore on disks that are not in the same chassis as the cluster nodes.

Worked example: a 5-node hyperconverged cluster

flowchart TB
  subgraph R1[Rack A]
    N1[pve-01]
    N2[pve-02]
    SW1[ToR-A]
  end
  subgraph R2[Rack B]
    N3[pve-03]
    N4[pve-04]
    SW2[ToR-B]
  end
  subgraph DR[DR site]
    N5[pve-05]
    PBS[PBS]
  end
  N1 --- SW1
  N2 --- SW1
  N3 --- SW2
  N4 --- SW2
  SW1 --- NET[Fabric]
  SW2 --- NET
  NET --- N5
  NET --- PBS
  N1 -. Ceph replication .-> N3
  N2 -. Ceph replication .-> N4

Failure analysis:

The operator as a failure domain

The most underrated failure domain is the operator. A change-management mistake applied to all nodes simultaneously takes down the entire cluster. Mitigations:

• Change windows that apply rolling changes to one node at a time.
• Pre-change snapshots of the cluster config (pmxcfs-tool versions + a tar of /etc/pve).
• Documented rollback in every change ticket.
• Two-person review for high-risk changes (firmware, kernel, cluster membership).

Common mistakes

• “Three servers in one rack is high availability.” No, it is not — the rack is a single failure domain.
• Backups in the same rack as the cluster. They share the rack, the power, and the network.
• Ignoring the operator. A solo admin applying a bad change to every node simultaneously is a real failure mode.

Key takeaways

• Failure domains are the units that fail together. Identify them, then ensure no single domain holds the entire critical path.
• The most common overlooked failure domain in Proxmox deployments is the network switch and the operator.
• Backups must live in a separate failure domain from the cluster they protect.

Knowledge check