When the node is never coming back

Intermediate⏱ ~18 minceph

What you'll learn

Read the cluster state in the first minutes of a whole-node loss
Distinguish what the cluster does automatically from what it waits for
Decide when a node is permanently lost rather than temporarily down
Quantify what leaving a dead node in the cluster costs

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

A node that will be back in an hour and a node that burned produce identical cluster output for the first ten minutes. Every decision after that depends on telling them apart, and the cluster cannot tell you.

The first ten minutes

ceph -s
ceph osd tree down
ceph health detail | head -20

  cluster:
    health: HEALTH_WARN
            8 osds down
            1 host (8 osds) down
            Degraded data redundancy: 4189234/12567702 objects degraded
              (33.333%), 1024 pgs degraded, 1024 pgs undersized
  services:
    osd: 48 osds: 40 up (since 4m), 48 in (since 3w)

Elapsed	Event	Mechanism
~20 s	peers report the OSDs down	heartbeat failure, `mon_osd_min_down_reporters`
immediate	affected PGs go `active+undersized+degraded`	one of three copies is missing
immediate	client IO continues	`min_size` 2 is still satisfied
~600 s	the OSDs are marked `out` — usually	`mon_osd_down_out_interval`
after `out`	backfill starts	CRUSH remaps those PGs elsewhere

ceph config get mon mon_osd_down_out_interval
ceph config get mon mon_osd_down_out_subtree_limit

Note the 48 osds: 40 up, 48 in line. Eight OSDs are unreachable and all forty-eight still count as cluster members holding placement.

What the cluster will not do for you

Down is a state. Weight is placement. A dead node keeps its full CRUSH
weight, so every PG still maps to it and nothing has moved.

# reweight is still 1.00000 on OSDs that cannot be contacted
ceph osd df tree | head -18

Recovery does not begin at the moment of failure. It begins when something sets those OSDs out, which is either the interval expiring or you deciding not to wait for it.

Declaring the loss permanent

Question	Answer that means permanent
Will these exact disks spin again?	no, or not knowably
Is the chassis powerable at all?	no
Is there a committed hardware date?	no, or beyond the exposure budget
Are the disks physically retrievable?	no — fire, theft, flood
Is the data on them still trusted?	no — water, heat, unknown writes

# what the node was carrying, from the last osdmap that saw it
ceph osd df tree | grep -A9 'host stor-07'

The declaration is an operational act, not a diagnosis. It says: we stop waiting, we accept a full re-replication, and we start it now.

What leaving it in costs

Consequence	Mechanism
PGs stay degraded	the third copy is never recreated
the mon store grows	osdmaps are not trimmed while PGs are unclean
scrubs do not run on those PGs	degraded PGs are skipped
the next failure is unbuffered	you are on two copies with no margin
capacity reporting is optimistic	`ceph df` still counts the dead OSDs

ceph pg ls degraded | head -5
du -sh /var/lib/ceph/*/mon.*/store.db 2>/dev/null

Quiz

Knowledge check · 4 questions

Q1. Why does a permanently dead node still hold placement immediately after it fails?

The monitors have not yet reached quorum on the failure, so the reports are recorded but not committed to a map `mon_osd_down_out_subtree_limit` defers the remap until an operator confirms the host will not be returning CRUSH maps by weight and rule, and liveness is not an input — only marking the OSDs out changes the mapping The placement group logs have not been trimmed, so the previous acting set is still recorded as authoritative
Q2. A whole host going down is marked out automatically under default settings, but a whole rack is not.

True False
Q3. Triage a storage node that has gone silent.

A 48-OSD cluster shows 8 osds down, one host, 33% of objects degraded. The data centre reports a fire suppression discharge in that rack and cannot say when the chassis will be assessed.
Q4. What does leaving a permanently dead node in the cluster cost, beyond the missing redundancy?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Decide permanence explicitly and record the time you decided; the exposure window is measured from that moment, not from the failure. Read mon_osd_down_out_subtree_limit before assuming the cluster will start recovery on its own, and mark the OSDs out yourself once the decision is made rather than letting a ten-minute timer own it.

Cross-course references

Kubernetes: a NotReady node keeps its pods scheduled until something evicts them
Linux: a failed device that is still in the array is not a spare, it is a hole