Patterns that help, and ones that only look like they do

Advanced⏱ ~18 minceph

What you'll learn

Apply device classes to separate competing workloads
Treat min_size 1 as an emergency action rather than a setting
Reject an osd failure domain as a substitute for a host
Keep the spare capacity physical on a small cluster

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Every small cluster eventually gets a proposal to make it look healthier, and two of the common ones convert a visible limitation into a silent one.

What genuinely helps

ceph osd crush class ls
ceph osd crush rule create-replicated fast default host ssd
ceph osd crush rule create-replicated bulk default host hdd
ceph osd pool set rbd crush_rule fast
ceph osd pool set backups crush_rule bulk

Separating a bulk pool from a latency-sensitive one matters more here than on a large cluster, because there are no other devices to absorb the contention when a backup job and a backfill overlap.

Pattern	Helps	Cost
Device classes splitting bulk from latency-sensitive pools	yes	planning time
A spare drive physically on the shelf	yes	one drive
Scoping `noout` to the host under work	yes	none
The monitor store on its own device	yes	one device per host
`min_size 1` as a standing setting	no	acknowledged writes with one copy
`osd` failure domain so size 3 fits	no	a host loss becomes data loss
EC k=2 m=1 for cold data	no	no redundancy at all while degraded
A fourth host	yes	a host

min_size 1 is an emergency action, not a configuration

ceph osd pool get rbd min_size
ceph osd pool set rbd min_size 1     # only to unblock, only briefly
ceph osd pool set rbd min_size 2     # immediately afterwards

At min_size 1 the cluster acknowledges a write that exists on one OSD. If that OSD is lost before the copy is made, the write is gone, and the surviving older copies cannot be identified as stale — the objects come back as unfound and someone has to choose between reverting and deleting.

Failure domain osd is not a substitute for a fourth host

# what not to do to make size 3 fit on three hosts
ceph osd crush rule create-replicated any default osd

This makes ceph -s report HEALTH_OK on three hosts, which is why it is proposed and why it is dangerous.

Keep the spare physical

ceph orch device ls
ceph orch apply osd --all-available-devices --unmanaged=true
ceph orch daemon add osd ceph-02:/dev/sdf

A small cluster has no spare capacity to absorb a drive failure, so the replacement drive is the recovery. Leave the OSD service unmanaged so an inserted spare is not consumed automatically the moment it appears, and add it deliberately when it is the drive you meant.

Quiz

Knowledge check · 4 questions

Q1. What is the consequence of setting the CRUSH failure domain to `osd` on a three-host cluster?

Recovery becomes faster, because every OSD in the cluster can now be a backfill target instead of only those on other hosts Nothing changes in practice, since CRUSH already descends to OSDs and there are three hosts to draw them from either way Roughly one PG in nine has all three copies on a single host, so a host loss makes those objects unavailable or lost The pool size must be raised to 4, because an osd domain needs one more copy to give the same protection as a host domain
Q2. Setting min_size to 1 is an acceptable standing configuration on a three-host cluster because it keeps writes flowing while a host is down.

True False
Q3. Review a proposal to make a three-host cluster more resilient.

A proposal contains three changes: switch the failure domain to `osd` so PGs stop showing undersized, set min_size to 1 so writes continue during maintenance, and move backups to an EC k=2 m=1 pool to free capacity.
Q4. Why does a device-class split matter more on a small cluster than on a large one?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Refuse changes whose main effect is a greener health status — an osd failure domain and a standing min_size of 1 both hide a real limitation rather than removing it. Keep a physical spare drive on site and leave the OSD service unmanaged, so an inserted device is added when you decide and not when it is detected.

Cross-course references

Kubernetes: anti-affinity rules that are preferred rather than required hide the same problem
Linux: silencing a monitoring check is not the same as fixing what it detected