CephXXIV · Replica Failure ScenariosReplica Failure Scenarios
One OSD fails on a size-3 pool
What you'll learn
- Sequence the cluster's response to a single OSD failure
- Explain the role of the down-out interval
- Predict client impact at each stage
- Decide when to intervene during a single-failure recovery
Prerequisites
None — start here.
Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18
Why this matters in production
Single OSD failures are routine — a cluster of a few hundred disks sees them monthly. The correct operator response is almost always “confirm and wait,” and knowing the exact sequence is what makes that confidence justified rather than lazy.
The timeline
t+0s — the OSD stops responding. Peers notice missed heartbeats.
t+20s — peers report it down. Enough peers agree
(mon_osd_min_down_reporters), and the monitors mark it down in a new
OSD map.
osd.47 downPGs whose acting set included osd.47 re-peer with the remaining members
and go active+undersized+degraded. Client I/O continues — two copies
meets min_size 2. There is a brief blip during peering, typically under
a second per PG.
t+600s — the down-out interval expires. mon_osd_down_out_interval
defaults to 600 seconds. The OSD is marked out, its CRUSH weight
effectively drops to zero, and CRUSH recomputes: the affected PGs get a
new third member.
osd.47 outPGs go active+undersized+degraded+remapped+backfilling.
t+600s to hours — backfill runs. The new OSDs receive the data. The degraded object count falls.
Completion — active+clean. Three copies again. Full redundancy
restored.
Why the ten-minute delay
The down-out interval exists because most OSD outages are transient — a daemon restart, a brief network hiccup, a kernel upgrade. Marking the OSD out immediately would trigger a full rebalance for an OSD that comes back in ninety seconds, and then a second rebalance when it returns.
Ten minutes is the default judgment about where that break-even sits. For planned work, override it explicitly:
ceph osd set noout # before maintenance
ceph osd unset noout # afterWhat clients see
| Stage | Client impact |
|---|---|
| detection (0–20s) | requests to the failed OSD time out and retry |
| peering | sub-second stall on affected PGs |
| degraded operation | none — full performance |
| backfill | some latency increase from background traffic |
| clean | none |
The only genuinely visible window is the first twenty seconds, and even that is retries rather than errors for well-behaved clients.
Quiz
Knowledge check · 4 questions
Q1. An OSD has been marked down but not out for the last five minutes. What is the state of the PGs it held?
Q2. Marking an OSD out is what triggers data movement, not marking it down.
Q3. Handle a single OSD failure during business hours.
At 10:15 on a Tuesday, osd.47 goes down on a 120-OSD size-3 cluster. Monitoring pages you. The cluster is at 68% capacity, all other OSDs are healthy, and the workload is at its daily peak.
Q4. Why does mon_osd_down_out_interval default to 600 seconds rather than zero?
Passing score: 75%. Answers are checked in this browser.
Production discipline
Alert on OSDs marked down, but set the escalation threshold above the down-out interval so a routine restart does not page anyone. Track time-to-clean as a metric: it is your real recovery window, and it is the number that determines how exposed a second failure makes you.
Cross-course references
- Kubernetes: node NotReady followed by pod eviction after a grace period is the same two-stage design
- Linux: a hot-spare rebuild in mdadm follows the same detect-then-commit pattern