Deciding whether to let recovery start

Intermediate⏱ ~18 minceph

What you'll learn

Explain what the down-out interval buys
Quantify the data a recovery would move
Choose between cluster-wide and per-OSD noout
Recognise when waiting stops being cheaper

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Recovery moves the failed host’s entire data footprint twice if the host comes back, and moves nothing if you guess the repair time correctly.

The timer

ceph config get mon mon_osd_down_out_interval        # 600
ceph config get mon mon_osd_down_out_subtree_limit   # rack

Elapsed	State
0–20s	peers report; monitors mark the OSDs down
20s–10min	down and in; degraded, no data movement
10min	monitors mark the OSDs out; backfill begins
after backfill	copies restored elsewhere; the host holds stale data

The subtree limit is `rack` by default, so a whole host is marked out
automatically and a whole rack is not.

What recovery would cost

ceph osd df tree
ceph osd ls-tree ceph-osd-07 > /tmp/lost-osds

ceph osd df --format json | python3 -c '
import sys,json
ids = set(int(x) for x in open("/tmp/lost-osds").read().split())
n = [o for o in json.load(sys.stdin)["nodes"] if o.get("device_class")]
used = sum(o["kb_used"]  for o in n if o["id"] in ids) / 1024**2
free = sum(o["kb_avail"] for o in n if o["id"] not in ids) / 1024**2
print("to re-replicate: %8.1f GiB" % used)
print("headroom left:   %8.1f GiB" % free)
print("survivors:       %8d OSDs" % len([o for o in n if o["id"] not in ids]))'

Choosing the scope

ceph osd set noout            # every OSD in the cluster
ceph osd unset noout

# only this host's OSDs
for id in $(ceph osd ls-tree ceph-osd-07); do ceph osd add-noout osd.$id; done
ceph osd dump | grep -c noout

for id in $(ceph osd ls-tree ceph-osd-07); do ceph osd rm-noout osd.$id; done

Situation	Decision
PSU swap or reboot, engineer on site	per-OSD noout, wait
Repair time unknown	no flag; let the interval run
Remaining tolerance is one failure domain	let recovery run regardless of repair time
Cluster above 75% used	assess capacity before allowing recovery
Host confirmed dead	no flag; recovery is the correct outcome
High write rate on the affected pools	waiting loses value quickly

When waiting stops paying

ceph config get osd osd_min_pg_log_entries   # 250
ceph config get osd osd_max_pg_log_entries   # 10000

A returning OSD replays the PG log if its copy is still within the
retained window. Past that, it copies whole PGs. The window is counted in
log entries, not in minutes.

watch -n 30 'ceph -s | grep -E "degraded|recovery|misplaced"'

Why the value of waiting decays with the write rate rather than with the clock

Each PG keeps a log of recent operations, bounded by osd_max_pg_log_entries rather than by time. When a returning OSD peers, its last log entry is compared against the authoritative log: if it is still inside the retained window, only the missing operations are replayed and the OSD is current in seconds. If the surviving replicas have written past the end of that window, the entire PG must be copied. An idle pool holds the window open for hours; a pool taking thousands of writes per second burns through ten thousand entries in well under a minute. The practical consequence is that noout is worth much more on a cold archive pool than on a busy RBD pool, and the ten-minute default is generous for one and already too long for the other.

Quiz

Knowledge check · 4 questions

Q1. What determines whether a returning OSD replays its PG log rather than copying whole PGs?

Whether the surviving replicas have written past `osd_max_pg_log_entries` since it left How many minutes it was absent, measured against `mon_osd_down_out_interval` Whether noout was set, since the flag preserves the retained log window for as long as it holds Whether the OSD was marked out, since an out OSD discards its local PG log
Q2. Setting noout cluster-wide also prevents an unrelated OSD elsewhere in the cluster from being marked out.

True False
Q3. Decide whether to set noout after a host failure.

A host with 12 OSDs holding 41 TiB raw is unreachable. The data centre reports a failed PSU with a replacement arriving in roughly four hours. The cluster is at 68% used and the pools take a steady 4000 writes per second.
Q4. Why is per-OSD noout preferable to the cluster-wide flag?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Decide the wait-or-recover question against repair time, remaining tolerance, and write rate — not against the ten-minute default alone. Scope the suppression with ceph osd add-noout rather than the global flag, and tie clearing it to the same change window that set it.

Cross-course references

Kubernetes: pod eviction timeouts make the same bet on how long a node will be absent
Linux: a global maintenance flag left set becomes the next incident