Host as the default failure domain

Why this matters in production

type host is what a Ceph cluster does if nobody makes a decision, and for the majority of clusters it is the correct decision. Knowing why — and knowing the specific cases where it is not enough — is the difference between a default you accepted and a default you chose.

What it protects against

A host is a genuine failure boundary: it shares a motherboard, PSUs, a kernel, an operating system, and network interfaces. Everything on it fails together for a long list of reasons.

rule replicated_rule {
  step take default
  step chooseleaf firstn 0 type host
  step emit
}

With size 3, this guarantees the three copies sit on three distinct hosts. Losing a host degrades PGs; it never makes them unavailable, as long as the pool has min_size 2 and at least three hosts remain.

The host count requirement

size 3, type host  →  minimum 3 hosts
size 4, type host  →  minimum 4 hosts

More precisely, the requirement is size hosts with usable OSDs. A host whose OSDs are all marked out does not count, which is why draining a host on a three-host cluster leaves PGs undersized.

Practical guidance is size + 1 hosts at minimum, so one host can be taken out for maintenance without leaving the cluster unable to satisfy its own rule.

When host separation is insufficient

• The cluster spans racks and a rack must survive. Host separation provides no rack protection; some PGs will have all copies in one rack.
• Hosts share a chassis. Blade and multi-node designs put several hosts behind one backplane and one pair of PSUs.
• Hosts share power or network. A single PDU or a single top-of-rack switch makes independent hosts fail together.

In each case the remedy is a rule at the appropriate level, which requires enough buckets at that level.

Verifying real placement

ceph pg map 7.3d
ceph osd find 12 | jq '.crush_location.host'
ceph osd find 47 | jq '.crush_location.host'
ceph osd find 83 | jq '.crush_location.host'

Sample several PGs and confirm the hosts differ. Reading the rule tells you what should happen; reading the placement tells you what does.

Quiz

Production discipline

Treat size + 1 hosts as the practical minimum rather than size, so routine maintenance does not leave PGs undersized. Verify by hand, once, that the hosts CRUSH names are genuinely distinct machines with distinct power and network — virtualised nodes, blades sharing a chassis, and shared PDUs all produce names that are not boundaries. And check real placement with ceph pg map and ceph osd find, because reading the rule tells you the intent and reading the acting set tells you the outcome.

Cross-course references

• Ceph: Part IV (Failure Domains) for identifying real boundaries.
• Ceph: Part XIV lesson on rack failure domains for the next level up.
• Ceph: Part CXXII (Small Cluster Risks) for clusters near the minimum.