Failure recovery in RADOS — what happens after an OSD goes away

Intermediate⏱ ~16 minceph

What you'll learn

Sequence the events from OSD failure to active+clean
Distinguish recovery from backfill and predict which will occur
Identify the throttles that control recovery pace
Balance recovery speed against client impact

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Recovery is the cluster spending its I/O budget on itself. Knowing the sequence tells you what to expect and when; knowing the throttles tells you what to do when it is too slow or too disruptive.

The sequence

t+0s     osd.7 stops responding to heartbeats
t+20s    peers report it after osd_heartbeat_grace
t+~20s   monitors mark it down; new osdmap published
         → PGs go active+undersized+degraded, still serving
t+600s   mon_osd_down_out_interval elapses; osd.7 marked out
         → CRUSH recomputes; PGs go active+remapped+backfilling
t+hours  backfill completes → active+clean

The important gap is between down and out. Down costs nothing; out starts data movement. Everything about maintenance flags exists to control that transition.

Recovery versus backfill

	Recovery	Backfill
Trigger	OSD returns; PG log covers the gap	new OSD, or gap exceeds the log
Work	copy only the missed objects	compare and copy the whole PG
Cost	proportional to changes	proportional to PG size
State	`recovering`	`backfilling`

A brief OSD restart produces recovery; replacing an OSD produces backfill. This is why a 30-second daemon restart is cheap and a hardware replacement is expensive, even though the OSD was absent in both cases.

noout is the flag for planned work; norebalance is for unplanned

Before a reboot or a daemon restart, ceph osd set noout prevents the 600-second timer from starting data movement. Unset it afterwards.

During an unplanned incident where recovery is competing with clients, ceph osd set norebalance stops the rebalancing portion while allowing degraded PGs to recover — which is usually what you want, because restoring redundancy matters more than optimal distribution.

ceph osd set noout            # planned maintenance
ceph osd set norebalance      # incident: prioritise redundancy
ceph osd unset noout

Both are visible in ceph status as a health warning, which is deliberate: a flag left set is itself a risk, and the warning is how you find it three weeks later.

The throttles

ceph config set osd osd_max_backfills 1
ceph config set osd osd_recovery_max_active 3
ceph config set osd osd_recovery_sleep 0.1
ceph config set osd osd_recovery_op_priority 3

Setting	Effect
`osd_max_backfills`	concurrent backfills per OSD
`osd_recovery_max_active`	concurrent recovery ops per OSD
`osd_recovery_sleep`	pause between recovery ops (seconds)
`osd_recovery_op_priority`	recovery priority relative to client I/O

Lowering these reduces client impact and lengthens the degraded window. That is the whole trade, and it is genuinely two-sided: a longer degraded window is a longer period with reduced redundancy.

Why recovery throughput is rarely limited by the throttles alone

Operators often raise osd_max_backfills and see little improvement, because the real constraint is elsewhere.

Backfill reads from surviving OSDs and writes to the target, so the rate is bounded by whichever is slower — and on an HDD cluster the reads compete with client I/O on the same spindles. Raising concurrency then increases queue depth without increasing throughput, and client latency rises while recovery does not speed up.

The other common constraint is the network: replicating a 16 TB OSD’s worth of data across a 10 GbE link has a floor of several hours no matter what the throttles say.

So the useful diagnostic before tuning is to identify the bottleneck — ceph osd perf for device saturation, interface counters for network — rather than raising limits and hoping. Recovery that is slow because the devices are saturated will not respond to more concurrency.

Watching it

ceph -s
ceph -w
ceph pg stat
ceph osd pool stats

ceph -s shows objects misplaced and degraded with a recovery rate. Watch the counts fall rather than the percentage — the percentage moves as the denominator changes and can mislead.

Quiz

Knowledge check · 4 questions

Q1. An OSD daemon is restarted and comes back in 30 seconds. What kind of data movement follows?

Recovery of only the operations it missed, because the PG logs cover the gap Full backfill, because leaving the acting set discards that OSD copy of the PG log No movement at all, because the OSD was never marked out and so kept its data current Backfill for the PGs where it was primary and recovery for those where it was a replica
Q2. On an HDD cluster, raising osd_max_backfills can leave recovery no faster while client latency gets measurably worse.

True False
Q3. Recovery after a failed OSD is severely impacting client latency. The team wants to slow it down. Advise on what to change and what it costs.

96-OSD HDD cluster with colocated WAL and DB. One 12 TB OSD failed and backfill has been running for two hours. Client p99 write latency has risen from 15 ms to 400 ms. Pool size 3, min_size 2, so PGs are degraded but serving. Estimated backfill completion is another six hours at current rate.
Q4. Explain the difference between noout and norebalance and when each is appropriate.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Set noout before planned work and unset it immediately afterwards — the health warning it raises exists so a forgotten flag is findable. During an incident, prefer norebalance over blanket throttling, since restoring redundancy outranks optimal distribution. And identify the bottleneck before adjusting recovery throttles: raising concurrency against saturated devices adds queue depth and client latency without adding throughput.

Cross-course references

Ceph: Part LVIII (Recovery) and Part LIX (Backfill) for each in depth.
Ceph: Part LX (Recovery Tuning) for the throttles in practice.
Ceph: Part XCV (Maintenance Flags) for the full flag set.