Maintenance with no spare failure domain

Intermediate⏱ ~17 minceph

What you'll learn

State what one host in maintenance costs for data and for quorum
Use the supported maintenance mode and its safety checks
Scope flags to a host rather than the cluster
Size the real length of a maintenance window

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

A reboot on a three-host cluster removes the last spare failure domain, and it removes it for the monitors and for the data at the same instant.

What one host in maintenance costs

Cluster	During one host down	A second failure then
3 hosts, 3 mons	quorum 2 of 3, PGs at 2 copies	quorum lost and PGs below min_size
4 hosts, 3 mons	quorum 2 of 3, PGs healing	quorum lost
5 hosts, 5 mons	quorum 4 of 5, PGs healing	quorum holds at 3 of 5

ceph -s
ceph mon stat
ceph quorum_status -f json | python3 -c '
import sys,json
d = json.load(sys.stdin)
print("mons:", len(d["monmap"]["mons"]), " in quorum:", len(d["quorum"]))'

Doing it the supported way

ceph -s | grep -q HEALTH_OK || echo "do not start"
ceph orch host maintenance enter ceph-02

cephadm inspects the daemons on the host before agreeing, and refuses when stopping them would break monitor quorum or remove the last instance of a service. --force overrides the daemon check; a separate confirmation is required where the move would take the monitors out of quorum. On three hosts you meet the first routinely and must never meet the second.

# once the host is back and its daemons are running
ceph orch host maintenance exit ceph-02
ceph -s
ceph orch ps --daemon-type osd --refresh

Scoping the flag rather than the cluster

ceph osd set-group noout ceph-02
ceph osd unset-group noout ceph-02
ceph osd dump | grep -E '^flags'

On three hosts noout prevents very little movement, because there is nowhere for data to move. What it prevents is OSD map churn and a second round of peering when the host returns. Scoped to the host, it also cannot be left set across the whole cluster for a month by accident.

The window is longer than the reboot

Stage	Typical
Shutdown and POST	3-8 minutes
Kernel or firmware application	2-20 minutes
OSDs starting and peering	1-5 minutes
Backfill of writes taken while down	minutes to hours
Return to HEALTH_OK	the sum of all of the above

Three hosts done serially at 25 minutes each is not 25 minutes of reduced tolerance, it is 75 — and the third host is the one that gets rushed because the window is running out.

Quiz

Knowledge check · 4 questions

Q1. What does `ceph orch host maintenance enter` do that stopping the daemons by hand does not?

It checks whether stopping the host daemons would break monitor quorum or remove the last instance of a service, and refuses unless overridden It migrates the data held on that host to the surviving hosts first, so the window starts from a fully replicated cluster rather than a degraded one It lowers the pool size for the duration of the window, so the absent copies do not register as degraded and no recovery is triggered It disables scrubbing across the cluster for the duration, so the survivors are not asked to serve deep scrub reads while short of a host
Q2. Setting `noout` before rebooting a host means the maintenance costs the cluster no redundancy.

True False
Q3. Plan a rolling kernel update across a three-host cluster.

Three hyperconverged hosts run three monitors and nine OSDs. A kernel CVE requires a reboot of each host. The change window is two hours on a weekday evening.
Q4. Why do the monitor margin and the data margin disappear at the same moment on a hyperconverged three-host cluster?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Never begin maintenance on a small cluster from anything other than HEALTH_OK, and return to it between hosts rather than at the end. Scope noout to the host under work with ceph osd set-group so a forgotten flag does not silently disable failure handling for the whole cluster.

Cross-course references

Kubernetes: draining a node from a three-node control plane removes the same two margins at once
Linux: serialised rolling reboots multiply the window, they do not share it