Peering after an OSD failure

Intermediate⏱ ~15 minceph

What you'll learn

Sequence the events from failure to restored redundancy
Distinguish this case from a restart
Estimate recovery duration for a failed OSD
Decide when to intervene rather than wait

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

A failed OSD is the most common Ceph incident, and the cluster handles it without intervention. Knowing the timeline tells you what to expect and when something has gone wrong.

The sequence

t+0      OSD stops responding
t+20s    peers report; monitors mark it down
         → its PGs: active+undersized+degraded, serving
t+600s   mon_osd_down_out_interval elapses; marked out
         → CRUSH recomputes; PGs: active+remapped+backfilling
t+hours  backfill completes → active+clean

The 600-second gap is the cluster giving the OSD a chance to return before committing to move terabytes.

ceph -s
ceph osd tree | grep -w osd.34
ceph health detail

Estimating duration

duration ≈ used capacity of the failed OSD / sustained backfill rate

For an 8 TB OSD at 60% full with a sustained rate of 300 MiB/s:

4.8 TB / 300 MiB/s ≈ 4.7 hours

The rate depends on the surviving devices, the network, the recovery throttles, and how much client I/O is competing.

ceph -s | grep recovery
ceph osd pool stats

Decide early whether the OSD is coming back

The 600-second window is a decision point, not just a delay.

If the OSD will return — a host rebooting, a transient fault — set noout and let it come back. Recovery from the PG log is far cheaper than backfill.

ceph osd set noout

If it will not — a failed drive, dead hardware — there is no benefit in waiting, and marking it out immediately starts recovery sooner:

ceph osd out 34

Both are reversible. The cost of guessing wrong is one unnecessary backfill in one direction, or ten extra minutes of degradation in the other, so the decision does not need to be agonised over — but it should be made rather than defaulted.

When to intervene

Wait if the degraded count is falling steadily and client impact is acceptable.

Intervene if:

the degraded count is static — recovery is blocked,
PGs are below min_size — those are not serving,
client latency is unacceptable — throttle recovery, accepting a longer window,
recovery is far slower than the estimate — investigate the bottleneck.

Why the surviving OSDs of the same batch are the risk during backfill

Backfill reads from every OSD that shares PGs with the failed one and writes to the OSDs taking over. That is sustained load on drives which, in a typical cluster, were purchased together and have accumulated similar wear.

So the period of reduced redundancy is also a period of elevated stress on the devices most likely to fail next. This is why failures cluster in time and why the independent-probability model overstates durability.

Two practical consequences. Throttling recovery reduces the stress but lengthens the degraded window, which is a genuine trade rather than a free improvement. And a second failure during backfill should be treated as more likely than the baseline rate suggests, which argues for having spares on site rather than ordering after the first failure.

ceph device ls
ceph osd perf | sort -k3 -n | tail -5

Verifying completion

ceph -s
ceph pg stat
ceph osd tree | grep -w osd.34         # should be out, or removed
ceph osd purge 34 --yes-i-really-mean-it   # once replaced

Quiz

Knowledge check · 4 questions

Q1. An OSD fails and will definitely not return — the drive is dead. What is the benefit of marking it out immediately?

It stops the monitors from continuing to probe an OSD that will never answer again It preserves the failed OSD PG logs so its replacement can recover rather than backfill It reduces the total volume of data that has to be moved to restore redundancy Recovery starts immediately rather than after the 600-second interval
Q2. Throttling recovery during a backfill is a free improvement to client latency.

True False
Q3. A 12 TB OSD at 70% full fails. Estimate recovery and decide whether to intervene.

96-OSD HDD cluster with BlueStore DB on NVMe. Pool size 3, min_size 2. The failed OSD held 8.4 TB. Backfill started ten minutes after the failure. ceph -s reports recovery at 180 MiB/s. Client p99 write latency has risen from 20 ms to 90 ms but no application is reporting errors. Spares are on site.
Q4. Give the timeline from OSD failure to restored redundancy and the decision point within it.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Treat the 600-second window as a decision point rather than a delay: noout if the OSD is returning, ceph osd out if it is not, and make the call rather than letting it default. Estimate recovery duration from used capacity over sustained rate so the degraded window is a number. Install replacements immediately so their backfill overlaps. And watch for a second failure specifically during recovery, since sustained load on same-batch drives makes it more likely than baseline.

Cross-course references

Ceph: Part LVII (Replacing Failed OSDs) for the replacement.
Ceph: Part LX (Recovery Tuning) for the throttle trade.
Ceph: Part IV (Failure Domains) for correlated failure.