Failure domains for Proxmox workloads

Why this matters in production

The default failure domain is host, which protects against a host failure and nothing larger. A rack, a switch, or a power feed taking out three hosts at once is a different event.

Choosing the domain

The requirement is absolute: failure_domain=rack with size=3 needs three racks, and two racks means PGs cannot be placed.

ceph osd tree | grep -cE '^-.*rack'
ceph osd pool get pve-vms size

Configuring CRUSH to match

# declare the physical topology
ceph osd crush add-bucket rack1 rack
ceph osd crush add-bucket rack2 rack
ceph osd crush add-bucket rack3 rack
ceph osd crush move rack1 root=default
ceph osd crush move rack2 root=default
ceph osd crush move rack3 root=default

# place hosts in racks
ceph osd crush move pve-01 rack=rack1
ceph osd crush move pve-02 rack=rack2
ceph osd crush move pve-03 rack=rack3

# a rule using the rack domain
ceph osd crush rule create-replicated rack-rule default rack
ceph osd pool set pve-vms crush_rule rack-rule

ceph osd tree

The tree must reflect the actual physical layout — a host recorded in the wrong rack silently defeats the protection.

Verifying the placement

# does any PG have two copies in one rack?
ceph pg dump pgs --format json 2>/dev/null | python3 -c '
import sys, json, subprocess
tree = json.loads(subprocess.check_output(["ceph","osd","tree","--format","json"]))
nodes = {n["id"]: n for n in tree["nodes"]}
rack = {}
def walk(nid, r=None):
    n = nodes.get(nid, {})
    if n.get("type") == "rack": r = n["name"]
    if n.get("type") == "osd": rack[n["id"]] = r
    for c in n.get("children", []): walk(c, r)
for n in tree["nodes"]:
    if n["type"] == "root": walk(n["id"])
d = json.load(sys.stdin)
bad = 0
for pg in d.get("pg_stats", []):
    racks = [rack.get(o) for o in pg.get("acting", []) if o >= 0]
    if len(set(racks)) < len(racks): bad += 1
print("PGs with two copies in one rack:", bad)'

# or test a rule without applying it
crushtool -i /tmp/crush.bin --test --show-mappings --rule 1 --num-rep 3 | head

Coordinating with Proxmox

Proxmox HA groups should align with the Ceph failure domains:

# /etc/pve/ha/groups.cfg
group: rack-aware
        nodes pve-01:2,pve-02:1,pve-03:1
        restricted 0

If Ceph tolerates a rack failure but Proxmox restarts all the affected VMs onto one surviving rack, that rack becomes a bottleneck at the moment the cluster is already recovering.

ha-manager groupconfig
ha-manager status

Quiz

Production discipline

Verify the CRUSH tree against the physical layout after every hardware move — CRUSH places by the map, and a host in the wrong rack silently defeats the protection. Balance host counts across failure domains; an uneven topology bounds pool capacity by the smallest domain.

Cross-course references

• Kubernetes: topology spread constraints must match real zone boundaries
• Linux: any redundancy scheme depends on the topology description being accurate