Reading the first ninety seconds of a partition

Intermediate⏱ ~18 minceph

What you'll learn

Recognise the partition symptom set in cluster status
Name the timers that generate each symptom and when
Distinguish a partition from a host or rack failure
Describe what the isolated side observes

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

A partition and a dead rack produce nearly the same ceph -s, and the two demand opposite first actions. Deciding which you have inside the first minute determines whether you spend the outage on a switch or on OSDs that never failed.

What the quorum side reports

ceph -s
ceph health detail

  cluster:
    health: HEALTH_ERR
            1/3 mons down, quorum ceph-mon-a,ceph-mon-b
            12 osds down
            1 host (12 osds) down
            Reduced data availability: 214 pgs inactive, 63 pgs peering
            Degraded data redundancy: 1841203/9204012 objects degraded

  services:
    mon: 3 daemons, quorum ceph-mon-a,ceph-mon-b (age 4m), out of quorum: ceph-mon-c
    osd: 36 osds: 24 up (since 90s), 36 in (since 3w)

Two numbers carry the diagnosis: 24 up ... 36 in, and the inactive PG count. Everything else in that output follows from them.

The timers that produce the symptom

Setting	Default	What it governs
`osd_heartbeat_interval`	6 s	how often peers ping each other
`osd_heartbeat_grace`	20 s	silence before a peer reports it down
`mon_osd_min_down_reporters`	2	reports needed before the mons act
`mon_osd_reporter_subtree_level`	`host`	reporters must sit in distinct subtrees
`mon_osd_down_out_interval`	600 s	down time before CRUSH removes it
`mon_osd_report_timeout`	900 s	mon silence before the mons act unilaterally

ceph config get osd osd_heartbeat_grace
ceph config get mon mon_osd_min_down_reporters
ceph config get mon mon_osd_reporter_subtree_level
ceph config get mon mon_osd_down_out_interval

The first report lands roughly 20 seconds after the link drops, the mons act once two reporters on distinct hosts agree, and CRUSH begins moving data ten minutes later. Nothing in that sequence distinguishes a partition from a power cut.

The signature that does distinguish it

Observation	Host failure	Partition
Daemon process on the down host	gone	running
Its admin socket	absent	answers
Its own log	ends abruptly	fills with heartbeat timeouts
Peers that time out	every peer	far-side peers only
Console or IPMI	host is off	host is up
`ceph -s` run on that host	not applicable	blocks, then times out

# on the host owning a supposedly down OSD
ceph daemon osd.42 status
tail -20 /var/log/ceph/$FSID/ceph-osd.42.log

2026-08-18T09:14:02 osd.42 heartbeat_check: no reply from 10.20.4.11:6802
  osd.7 since 2026-08-18T09:13:38 (oldest deadline 2026-08-18T09:13:58)

A scatter of unrelated peers means the OSD is sick. A set of peers that all share a rack, a switch, or a subnet means the network is.

What the far side reports

Nothing — and that is itself the evidence. On an isolated host the ceph client blocks trying to reach a monitor it cannot see, then returns a connection error. Its daemons are healthy, their logs full of outbound heartbeat failures aimed at the quorum side, and their view of who is down is the exact mirror of the quorum side’s. That mirror changes nothing: no report from that side reaches a monitor allowed to act on it.

Quiz

Knowledge check · 4 questions

Q1. During a partition, why does the OSD down list describe the far side rather than the failed component?

The monitors probe every OSD directly and record the ones whose probes time out within `mon_osd_report_timeout` seconds CRUSH recomputes placement as soon as a host stops answering, and excludes every unreachable host from the result Down-marking is driven by peer reports delivered to the quorum, and only the quorum side can commit an OSDMap change The OSDs on the far side detect the isolation and mark themselves down so that the quorum side does not have to
Q2. An OSD whose admin socket still answers on its own host can nevertheless be listed as down by the cluster.

True False
Q3. Classify an unfolding incident before taking action.

At 09:14 the cluster reports 12 of 36 OSDs down, all on hosts in rack B, plus one monitor out of quorum. A junior engineer proposes marking the 12 OSDs out to start recovery.
Q4. Why does an isolated host see a mirror image of the cluster status, and why is it harmless?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Never infer hardware health from ceph osd tree during a suspected network event — the down list is a hearing record, not a failure record. Confirm liveness out of band on one supposedly down host before you touch the OSDMap, and set noout first so the ten-minute clock does not start a rebalance you will spend the rest of the day reversing.

Cross-course references

Kubernetes: a NotReady node is a report of missed kubelet heartbeats, not of a dead machine
Linux: a timeout tells you the reply did not arrive, never why