Node failure — "did not answer in time" is the only signal you get

Why this matters in production

You send a request to a node. Nothing comes back. There are three possibilities and no way to tell them apart from where you are standing: the node is dead, the node is alive but slow, or the network dropped your message.

This is not an engineering limitation to be solved. It is fundamental, and every failure-handling design is a choice about how long to wait before guessing.

How Ceph guesses

OSDs exchange heartbeats with the peers they share placement groups with:

osd_heartbeat_interval        6s    how often heartbeats are sent
osd_heartbeat_grace          20s    silence before a peer is reported failed
mon_osd_min_down_reporters    2     distinct reporters needed to act
mon_osd_down_out_interval   600s    down duration before marked out

The sequence:

1. osd.7 stops answering heartbeats from its peers.
2. After osd_heartbeat_grace, peers report it to the monitors.
3. Once enough distinct reporters agree, monitors mark it down and publish a new map.
4. If it is still down after mon_osd_down_out_interval, monitors mark it out, CRUSH recomputes without it, and recovery begins.

Why multiple reporters are required

mon_osd_min_down_reporters defaults to 2. A single OSD reporting a peer as failed proves only that those two cannot reach each other, which may be the reporter’s problem.

Requiring several distinct reporters distinguishes “osd.7 is down” from “osd.3 has a broken NIC and cannot reach anyone”. On clusters with few hosts this matters, and it is why the setting is tied to reporters from different hosts rather than merely different OSDs.

Tuning, and when not to

The defaults are conservative and suit most clusters. Reasons to change them are narrow:

• Raise osd_heartbeat_grace only where network jitter is confirmed and understood — raising it delays every real failure detection too.
• Lower mon_osd_down_out_interval on clusters where recovery is cheap and fast (small, all-NVMe) and long degradation is the bigger risk.
• Raise it on large clusters where re-replicating an OSD is expensive and reboots are routine.

The instinct to tune these after an incident is usually wrong. Flapping is a symptom of a network or hardware problem, and widening the grace window hides it rather than fixing it.

Quiz

Production discipline

Set noout before any planned maintenance, because it blocks the expensive out decision while leaving failure detection working. Resist tuning heartbeat and grace settings in response to an incident: flapping is a symptom of a network or hardware fault, and widening the window delays detection of every genuine failure too. Treat ceph osd perf outliers as actionable without a health warning, since the failure mode that health checks cannot see — a slow but responsive OSD — is the one that degrades the most clients.

Cross-course references

• Ceph: Part LV (OSD States) for the full state machine.
• Ceph: Part LXXIX (Slow Ops) for diagnosing the slow-OSD case.
• Ceph: Part XCV (Maintenance Flags) for noout and its relatives.