How the cluster learns a host is gone

Intermediate⏱ ~18 minceph

What you'll learn

Trace how an OSD is marked down
Distinguish the down state from the out state
Read the degraded object accounting
Identify which client I/O is actually blocked

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

The first minute after a host disappears determines what you are allowed to do next, and almost every wrong decision comes from misreading it.

Detection

OSDs heartbeat their peers, not the monitors. A monitor marks an OSD down only when enough other OSDs say so, and the threshold has a topology condition attached to it.

ceph config get osd  osd_heartbeat_interval          # 6
ceph config get osd  osd_heartbeat_grace             # 20
ceph config get mon  mon_osd_min_down_reporters      # 2
ceph config get mon  mon_osd_reporter_subtree_level  # host
ceph config get mon  mon_osd_report_timeout          # 900

Setting	Effect
`osd_heartbeat_interval`	how often peers ping each other
`osd_heartbeat_grace`	silence tolerated before a peer files a report
`mon_osd_min_down_reporters`	reports needed to act
`mon_osd_reporter_subtree_level`	reports must come from distinct buckets at this level
`mon_osd_report_timeout`	the monitor marks an OSD down unilaterally after this long with no contact

ceph osd tree down
ceph health detail | grep -E 'OSD_DOWN|OSD_HOST_DOWN|PG_DEGRADED'

ID  CLASS  WEIGHT    TYPE NAME             STATUS
-7         87.31982  host ceph-osd-07
18   ssd    7.27664      osd.18             down
19   ssd    7.27664      osd.19             down
20   ssd    7.27664      osd.20             down

Down is not out

ceph osd dump | grep -E '^osd[.](18|19|20) '

State	Meaning	Effect on placement
`up`, `in`	serving	normal
`down`, `in`	not serving, still mapped	PGs degraded; no data moves
`down`, `out`	not serving, unmapped	PGs remap; backfill starts
`up`, `out`	serving but unmapped	drains only

While the OSDs are down and in, the cluster is waiting. Nothing is being
copied and nothing is being lost. That window is the whole decision.

Reading the accounting

ceph -s
ceph pg stat

  data:
    pools:   9 pools, 4097 pgs
    objects: 41.20M objects, 137 TiB
    pgs:     3721 active+clean
             376  active+undersized+degraded
    degraded: 3.71M/123.60M objects degraded (3.001%)

Counter	Question it answers
`degraded` objects	how many copies are missing
`active+undersized`	PGs serving with a short acting set
`peered` without `active`	PGs below `min_size`, not serving
`undersized` PG count	the blast radius of a second failure

Which I/O is blocked

ceph pg dump_stuck inactive
ceph health detail | grep -E 'PG_AVAILABILITY|SLOW_OPS'

# a degraded PG still serves; a peered one does not
ceph pg ls degraded | head -5
ceph pg ls peered  | head -5

Quiz

Knowledge check · 4 questions

Q1. What must be true before monitors mark an OSD down on peer reports?

The monitor must fail to reach the OSD directly on the public network, because reports from peers on their own are only advisory The OSD must stop answering clients for longer than `osd_heartbeat_grace`, which the monitors measure themselves Reports must arrive from at least `mon_osd_min_down_reporters` OSDs in distinct buckets at `mon_osd_reporter_subtree_level` A quorum of monitors must each observe the failure independently before the leader commits the new osdmap epoch
Q2. Monitors detect a failed OSD by heartbeating it directly.

True False
Q3. Triage the first minutes of a host loss.

A 12-host cluster loses contact with ceph-osd-07 and its 12 OSDs. Health shows 376 PGs active+undersized+degraded and 3% of objects degraded.
Q4. What does an OSD in the down and in state mean for data placement?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Read ceph osd tree down before anything else — a failure confined to one host bucket is a different problem from OSDs scattered across hosts. Separate the degraded count from the inactive count; degraded means the cluster is waiting, inactive means users are already affected.

Cross-course references

Kubernetes: node NotReady is a report from the kubelet, not an observation by the API server
Linux: agreement between two interfaces on one machine is one opinion, not two