CephLVI · OSD FailureOSD Failure
Deciding what to do about a failed OSD
What you'll learn
- Enumerate the options after an OSD failure
- Weigh the cost of each against the situation
- Choose deliberately rather than by timer
- Communicate the decision and its consequences
Prerequisites
None — start here.
Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18
Why this matters in production
Doing nothing is a decision, and it is the one the down-out interval makes for you. Choosing deliberately means the cluster does what you intended rather than what the default timer produced.
The four options
| Option | When | Cost |
|---|---|---|
| Restore the OSD | daemon crashed, device healthy | recovery delta, minutes |
| Wait and let it go out | device failed, capacity available | full rebalance |
| Mark out immediately | device failed, want the rebalance to start now | full rebalance, sooner |
| Hold with noout | investigating, or a short planned absence | reduced redundancy for the duration |
Restore
# HOST is the node running osd.13, from `ceph osd find 13`; DEV is that OSD's
# block device on it. Substitute your own:
HOST=ceph-04
DEV=/dev/sdc
ssh "$HOST" smartctl -H "$DEV"
ceph orch daemon restart osd.13
ceph -sViable when the device is healthy. Recovery is a delta from the PG log if
the absence was short, and the cluster returns to active+clean quickly.
Wait
The default. The interval expires, the OSD is marked out, and the rebalance begins. Appropriate when the device has clearly failed and there is nothing to decide.
Mark out immediately
ceph osd out 13Starts the rebalance now rather than in ten minutes. Worth doing when the device has definitively failed and you want redundancy restored as early as possible — ten minutes of a size-3 pool at two copies is ten minutes of exposure.
Hold with noout
ceph osd add-noout 13Keeps the cluster degraded rather than rebalancing. Appropriate for a short investigation or a planned absence, and inappropriate for anything measured in days.
The decision inputs
| Question | Bearing |
|---|---|
| Is the device healthy? | determines whether restoring is possible |
| How long until the OSD can return? | minutes favours holding, days favours rebalancing |
| Is there capacity for the rebalance? | check before committing to it |
Is the pool at min_size anywhere? | urgency rises sharply |
| Is other maintenance in progress? | avoid concurrent rebalances |
ceph osd df # capacity for the rebalance
ceph pg dump_stuck undersized # how exposed the pool is
ceph -s # other recovery in progressQuiz
Knowledge check · 4 questions
Q1. What should be checked before marking a failed OSD out?
Q2. Marking a failed OSD out immediately shortens a multi-hour exposure window by only ten minutes.
Q3. Choose a response to a failed OSD.
osd.13 is down. SMART reports the device as failing with reallocated sectors climbing. The cluster is at 81% capacity with 200 OSDs. Another OSD failed two days ago and its rebalance completed yesterday.
Q4. When is holding an OSD with noout inappropriate?
Passing score: 75%. Answers are checked in this browser.
Production discipline
Decide deliberately rather than letting the down-out interval decide; check device health and cluster capacity, then choose. Verify capacity before allowing a rebalance on a cluster above 80% — the redistribution can stall and leave the cluster worse than the failure did.
Cross-course references
- Kubernetes: deciding whether to let a node’s pods be evicted or to fix the node first is the same call
- Linux: whether to rebuild a RAID array or repair the member first involves identical reasoning