Headroom for maintenance

Why this matters in production

Maintenance takes capacity out of service, and the cluster must run without it for the duration. Whether that is a problem depends on headroom that is easy not to check.

What each maintenance mode costs

The noout distinction is the largest single lever: rebooting a host without it triggers a full drain of its OSDs and a refill when they return, moving that host’s data twice for a ten-minute reboot.

ceph osd set noout
# reboot
ceph osd unset noout

Better still, for a host:

ceph orch host maintenance enter ceph-osd-03
# reboot
ceph orch host maintenance exit ceph-osd-03

Planning against headroom

# what is offline during the window?
ceph osd df tree | awk '/host/ {h=$NF} /osd\./ {s[h]+=$5} END {for(k in s) print k, s[k]}'

# does the cluster function without it?
ceph osd pool get rbd-vms min_size
ceph osd tree | grep -c 'host '

With size=3, min_size=2, and one host offline, PGs are undersized but still serve I/O — provided the remaining host count is at least min_size. That is the check.

Not overlapping with reduced redundancy

# before starting any maintenance
ceph -s
ceph health detail | grep -E 'PG_DEGRADED|OSD_DOWN|PG_BACKFILL_FULL'

Starting a rolling upgrade while a PG is already degraded means a further failure during the window takes it below min_size. The rule that avoids this is simple:

begin maintenance only from HEALTH_OK

ceph health | grep -q HEALTH_OK && echo 'safe to proceed' || echo 'resolve first'

The upgrade case specifically

ceph orch upgrade start --ceph-version 20.2.1
ceph orch upgrade status
watch -n 30 'ceph -s'

cephadm upgrades one daemon at a time and waits for health between steps, which handles the sequencing. What it does not do is create headroom — a cluster too full to tolerate one host being briefly unavailable is still too full during an upgrade.

Quiz

Production discipline

Use ceph orch host maintenance enter rather than a plain reboot for planned host work; without it the host’s data moves twice for a ten-minute outage. Begin any maintenance only from HEALTH_OK — overlapping a maintenance reduction with an existing one is what turns a routine window into an incident.

Cross-course references

• Kubernetes: cordon and drain versus an abrupt node restart is the same distinction
• Linux: setting a RAID array to not auto-rebuild during planned member removal