What the sixth and seventh hosts buy

Advanced⏱ ~18 minceph

What you'll learn

Identify the host count at which backfill replaces hardware repair
Apply the placement and healing host counts for erasure coding
Promote a failure domain from host to rack safely
Use device classes to separate workloads

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

The step from three hosts to six changes the recovery mechanism from hardware logistics to backfill, and that is the whole argument.

The self-healing threshold

Hosts	Size 3 replicated	Erasure coding available
3	works; a host loss is not repairable	2+1 only, also not repairable
4	a host loss backfills onto the fourth	2+1, repairable
6	comfortable headroom	4+2 placeable, not repairable
7	headroom for two losses	4+2 repairable
9	rack failure domains become possible	6+3 placeable

ceph osd tree
ceph osd crush rule dump
ceph osd erasure-code-profile ls

Recovery time also falls with host count. A failed host on six is rebuilt
from five sources onto five targets; on twelve, from eleven onto eleven.
The parallelism is in both directions.

ceph config set osd osd_max_backfills 2
ceph config set osd osd_recovery_max_active 4
ceph -s | grep -E 'recovery|backfill'

Erasure coding, placed versus healed

ceph osd erasure-code-profile set ec42 k=4 m=2 \
  crush-failure-domain=host crush-device-class=hdd
ceph osd erasure-code-profile get ec42
ceph osd pool create archive erasure ec42

Profile	Hosts to place	Hosts to heal a host loss	Raw per logical TiB
2+1	3	4	1.50
4+2	6	7	1.50
6+3	9	10	1.50
8+3	11	12	1.38
Size 3 replicated	3	4	3.00

Promoting the failure domain to rack

ceph osd crush add-bucket rack1 rack
ceph osd crush add-bucket rack2 rack
ceph osd crush add-bucket rack3 rack
ceph osd crush move rack1 root=default
ceph osd crush move ceph-01 rack=rack1
ceph osd crush move ceph-02 rack=rack1

ceph osd crush rule create-replicated by-rack default rack
ceph osd crush tree
ceph osd pool set vms crush_rule by-rack

Requirement	Value
Racks needed for size 3	3
Capacity per rack	at least a third of the pool
Hosts per rack	enough that a rack loss leaves the pool servable
Data movement on the change	large; throttle and schedule it

Device classes

ceph osd crush class ls
ceph osd crush class ls-osd nvme
ceph osd crush rule create-replicated fast default host nvme
ceph osd pool set vms crush_rule fast

Why healing an erasure-coded pool needs one more host than placing it

CRUSH will not place two shards of the same placement group on the same failure domain, because doing so would make one host loss cost two shards. With a 4+2 profile on exactly six hosts, every host already holds one shard of every placement group, so when a host is lost there is no host without a shard to receive the replacement — the placement group stays degraded until the host returns, exactly as on a three-host replicated cluster. Adding a seventh host gives CRUSH a candidate with no existing shard, and the lost shards regenerate in minutes. The general rule is that placement needs k plus m hosts and self-healing needs k plus m plus one, and the difference between those two numbers is the difference between a cluster that repairs itself and one that waits for a technician.

Quiz

Knowledge check · 4 questions

Q1. A 4+2 erasure-coded pool runs on exactly six hosts. One host fails permanently. What happens?

The pool loses data, since a 4+2 profile tolerates the loss of only one shard before reconstruction becomes impossible Five shards are still enough to read, so the pool re-encodes itself to a 4+1 profile until a replacement host is added The placement groups stay degraded, because every surviving host already holds a shard and CRUSH has no candidate for the replacement The pool becomes read-only until the host is replaced, because new writes cannot be encoded while fewer than six hosts are available
Q2. Recovery from a host failure completes faster on a twelve-host cluster than on a six-host cluster with the same total capacity.

True False
Q3. Move a populated pool to a rack failure domain.

A 9-host cluster in three racks holds a 300 TiB replicated pool at 71% utilisation with a host failure domain. The requirement is now to survive a rack loss.
Q4. How many hosts does a 6+3 erasure-coded pool need to place its shards, and how many to heal a host loss?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Size erasure-coded pools to k plus m plus one hosts, not k plus m — the difference decides whether a host loss is repaired by backfill or by a technician. Check rack capacity balance before promoting a failure domain; the smallest rack caps the pool, and the movement itself is days of backfill that cannot be cleanly paused part way.

Cross-course references

Kubernetes: a scheduler with no spare node cannot reschedule what a node loss evicted
Linux: rebuild parallelism scales with the number of surviving members, in both directions