Sizing for one host down, or for two

Why this matters in production

The reserve is a line on the purchase order, and its size is decided by how many failure domains you intend to survive rather than by how full you are.

The arithmetic of the reserve

Spread evenly, each host holds one Nth of usable capacity. Surviving k simultaneous losses means keeping k Nths free.

Hosts are rarely identical, so the requirement is the largest one.

ceph osd df tree -f json | python3 -c '
import sys,json
d = json.load(sys.stdin)
h = [(n["name"], n.get("kb_used",0)/1024**3)
     for n in d["nodes"] if n.get("type") == "host"]
h.sort(key=lambda x: -x[1])
for n, t in h:
    print("%-16s %8.1f TiB raw used" % (n, t))'

When the host is the wrong unit

ceph osd crush rule dump | grep -E '"type"|"op"'
ceph osd tree | grep -E 'rack|chassis|host'

If the CRUSH failure domain is rack and each rack holds four hosts, the reserve is a rack — four times the number you were about to write down.

What the second host costs

At six hosts the second level of tolerance adds a quarter to the cost of every usable byte. At twenty it adds a few per cent. Sizing for two simultaneous losses is a decision that becomes affordable somewhere around twelve hosts and is punitive below eight.

Choosing, and recording the choice

cat >> /secure/ceph-capacity-plan.txt <<'EOF'
failure tolerance sized for: 1 host
rationale: 6 hosts, 2 racks, no shared PDU within a rack
revisit at: 12 hosts, where the second host of reserve costs 8 percent
EOF

Quiz

Production discipline

Size the reserve against the largest thing that can fail at once, which is often a rack or a chassis rather than a host. Record the physical rationale beside the number, and set a cluster size at which the decision to reserve for two failures should be revisited.

Cross-course references

• Kubernetes: pod anti-affinity across nodes is worthless if the nodes share a hypervisor
• Linux: RAID spare counts are chosen against correlated failure, not independent probability