Reboots and the down-out interval

Intermediate⏱ ~17 mincephsystemctl

What you'll learn

Choose between noout and draining for a given outage
Compute the decision from the down-out interval
Execute a reboot correctly
Handle a reboot that takes longer than planned

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

A reboot and a decommission look similar and need opposite handling.

The decision

ceph config get osd mon_osd_down_out_interval

outage shorter than the interval
  → the OSDs return before anything happens automatically
  → noout is belt-and-braces, not strictly needed
  → but it protects against the interval being reached

outage longer than the interval
  → the OSDs would be marked out and recovery would start
  → noout prevents that, if they are returning
  → draining is correct if they are not

Outage	Approach
A reboot, minutes	`noout` on the host
A kernel upgrade, tens of minutes	`noout`, or maintenance mode
Hardware work, hours	maintenance mode
Days	drain properly
Permanent	drain and remove

Why not drain for a reboot

Draining a host for a ten-minute reboot:
  moves the host's entire contents away
  then moves it all back when the host returns
  two full rebalances for ten minutes of downtime

ceph osd df tree | grep -A15 ceph-03

On a twelve-drive host holding 18 TB, that is 36 TB of movement for a reboot. noout is the difference between that and none.

Executing a reboot

# 1. cluster healthy
ceph -s | grep HEALTH_OK

# 2. scoped suppression
ceph osd add-noout ceph-03

# 3. reboot
ssh ceph-03 systemctl reboot

# 4. wait and verify
while ! ceph osd tree | grep -A15 ceph-03 | grep -q 'up'; do sleep 15; done
ceph osd tree | grep -A15 ceph-03

# 5. clear
ceph osd rm-noout ceph-03
ceph -s

# using maintenance mode, which handles the sequence
ceph orch host maintenance enter ceph-03
ssh ceph-03 systemctl reboot
ceph orch host maintenance exit ceph-03

When a reboot takes longer than planned

ceph -s
ceph osd tree | grep -A15 ceph-03

Situation	Action
Booting slowly, will return	wait; `noout` is holding
Failed to boot, being fixed	keep `noout` if the fix is short
Failed to boot, unclear	clear `noout` and let recovery start
Hardware failed	clear `noout`, drain properly

The decision point: is the host returning within a time comparable to the
recovery duration? If not, the recovery should start.

ceph osd rm-noout ceph-03
# the OSDs are marked out after the interval, and recovery begins

Quiz

Knowledge check · 4 questions

Q1. Why does a returning OSD after a short reboot catch up cheaply?

Its last log position is still in the primary's PG log, so the primary sends exactly the missed operations rather than the full contents It kept its objects in the BlueStore cache across the reboot, so the primary need only confirm that the object versions still match its own copies The monitors replay the writes that occurred during the outage, from the map history they retain for exactly this Nothing changed while it was down, because the primary holds writes to a PG until its whole acting set is present
Q2. Draining a host is the safe approach for any maintenance including a reboot.

True False
Q3. Handle a reboot that has overrun.

A host was rebooted with noout set 90 minutes ago and has not returned. The cause is being investigated and the timeline is unclear.
Q4. What is the decision point between keeping `noout` and letting recovery start?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Use noout for a reboot rather than draining — a drain moves the host’s contents away and back for an outage measured in minutes. When an outage overruns and the timeline is unclear, clear the flag: degradation with no recovery is worse than the backfill the recovery costs.

Cross-course references

Kubernetes: cordoning for a reboot rather than draining avoids the same wasted churn
Linux: temporarily failing an array member differs from removing it