What three and four hosts actually tolerate

Why this matters in production

The sentence people say is that the cluster tolerates a node failure. The useful question is how many, of what, and whether it heals afterwards.

Data: what one and two hosts down actually do

A size 3, min_size 2 pool serves normally with two copies. Below min_size the PG does not go active, and the client sees I/O hang.

The share is the chance that both failed hosts were among the three that a PG mapped to, so it falls quickly with host count and never reaches zero.

ceph osd pool get rbd size
ceph osd pool get rbd min_size
ceph pg ls-by-pool rbd | awk 'NR>1 {print $10}' | sort | uniq -c

Quorum is a separate count

ceph mon stat
ceph quorum_status -f json | python3 -c '
import sys,json
d = json.load(sys.stdin)
n = len(d["monmap"]["mons"])
print("mons %d  quorum needs %d  tolerates %d" % (n, n//2+1, n-(n//2+1)))'

On a hyperconverged small cluster the monitors sit on the same chassis as the OSDs, so both counts are consumed by the same failure.

The fourth host changes recovery, not the count

Four hosts still tolerate one simultaneous host loss. What changes is that the loss heals: CRUSH now has a domain that does not already hold a copy, so the cluster returns to three copies without anyone restoring hardware.

ceph osd getcrushmap -o /tmp/cm
crushtool -i /tmp/cm --test --rule 0 --num-rep 3 --show-bad-mappings
crushtool -i /tmp/cm --test --rule 0 --num-rep 4 --show-bad-mappings

The second run reports bad mappings for every PG on a three-host cluster, which is CRUSH stating plainly that the rule cannot produce four distinct hosts from three.

Checking your own numbers

ceph osd crush rule dump | grep -E 'type|op'
ceph osd tree | grep -c '^.* host '
ceph health detail

Quiz

Production discipline

Write the tolerance down as two separate numbers — hosts tolerated for data, and monitors tolerated for quorum — and note whether recovery follows automatically. On a hyperconverged small cluster both numbers are consumed by the same failure, which is the fact most design documents leave out.

Cross-course references

• Kubernetes: an etcd quorum of three tolerates one member loss for the same reason
• Linux: majority-based clustering rejects even member counts everywhere it appears