Heartbeats — how OSDs detect each other and report failure

Intermediate⏱ ~15 minceph

What you'll learn

Describe the heartbeat exchange between OSDs
Identify which network heartbeats traverse
Explain why a slow OSD passes heartbeats
Interpret heartbeat-related log messages

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Heartbeats decide whether an OSD is considered alive, and they are deliberately cheap — which means they answer “is the process responding?” and not “is this OSD useful?”. Most confusing OSD incidents live in the gap between those two questions.

The mechanism

Each OSD exchanges heartbeats with the peers it shares PGs with, not with every OSD in the cluster:

osd_heartbeat_interval    6s     send interval
osd_heartbeat_grace      20s     silence before reporting a peer failed
osd_heartbeat_min_peers   10     minimum peers to heartbeat with

Heartbeats travel on both the public and cluster networks where both are configured, using dedicated ports:

ceph osd find 12
# shows "heartbeat_back_addr" and "heartbeat_front_addr"
ss -tlnp | grep ceph-osd

The dual-network heartbeat is deliberate: an OSD reachable on one network and not the other is detected as partially failed rather than appearing healthy.

What happens on failure

1. osd.7 stops replying to heartbeats from its peers
2. After osd_heartbeat_grace, peers report to the monitors
3. Monitors require mon_osd_min_down_reporters distinct reporters
4. Monitors mark osd.7 down and publish a new map

A saturated OSD still answers heartbeats

Heartbeat messages are tiny and are handled on a separate path from client operations. An OSD whose device is saturated, whose queues are full, and which is taking seconds to complete client writes will still reply to heartbeats within milliseconds.

The cluster therefore considers it entirely healthy while it delays every write in every PG it participates in. This is the single most common “Ceph is slow but healthy” scenario.

Detection requires looking at latency rather than liveness:

ceph osd perf                       # commit and apply latency per OSD
ceph daemon osd.7 dump_ops_in_flight
ceph health detail | grep -i slow

ceph osd perf outliers deserve investigation on their own, with no health warning required.

Log messages worth recognising

Message	Meaning
`heartbeat_check: no reply from osd.N`	this OSD cannot reach osd.N
`wrongly marked me down`	the OSD was alive and got marked down — network or overload
`osd.N reported failed by osd.M`	a report reached the monitors
`failed to bind to heartbeat address`	configuration or port conflict at startup

wrongly marked me down is the important one. It means the OSD was running and could not convince its peers, which points at the network between them or at the OSD being too busy to respond in time — the latter being a genuine overload signal even though the OSD is “alive”.

Why heartbeat tuning is usually the wrong response to flapping

When OSDs flap, raising osd_heartbeat_grace makes the flapping stop. It does so by declining to notice the condition that caused it.

The underlying causes are a small set: a degrading network link, a switch dropping frames, an overloaded host whose OSDs cannot schedule their heartbeat threads, or a device so saturated that the OSD process stalls. Each of these is a real fault with a real fix.

Widening the grace window delays detection of every genuine failure by the same amount, so the cost is paid during the next real outage. It is occasionally correct — on a cluster with known, measured, and irreducible network jitter — but it should follow a measurement rather than precede one.

The diagnostic order is: check interface error counters on the hosts involved, check ceph osd perf for the flapping OSDs, check host load and memory, and only then consider timing.

Verifying the heartbeat path

On a cluster with separate public and cluster networks, both paths must work:

CLUSTER_NET_IP=10.20.0.11
PUBLIC_NET_IP=10.20.0.11
ceph osd find 12 | jq '.osd_fault_injection, .addrs'
# from another OSD host, test both networks
ping -M do -s 1472 ${CLUSTER_NET_IP}
ping -M do -s 1472 ${PUBLIC_NET_IP}

A cluster network that works for replication but drops heartbeat traffic — because of a firewall rule covering only the replication ports — produces OSDs that replicate fine and are repeatedly marked down.

Quiz

Knowledge check · 4 questions

Q1. Why does an OSD with a saturated device continue to pass heartbeats?

Heartbeats are answered by the kernel's TCP stack rather than by the OSD process itself The monitors discard failure reports about an OSD whose reported utilisation is already high Heartbeat messages are tiny and handled separately from client operations Heartbeats are exchanged only between peers that currently share no in-flight client work
Q2. An OSD too busy to answer its heartbeats within the grace window is marked down even though its process is running normally.

True False
Q3. OSDs on a cluster with separate public and cluster networks replicate successfully but are repeatedly marked down. Diagnose.

24 OSD hosts with public_network on 10.0.1.0/24 and cluster_network on 10.0.2.0/24. Replication traffic flows normally and recovery completes. OSDs are intermittently marked down and come back within seconds. Logs show "heartbeat_check: no reply from osd.N back addr" repeatedly. A firewall was recently deployed on the cluster network with rules for the OSD data ports.
Q4. Explain why heartbeats traverse both the public and cluster networks when both are configured.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Treat heartbeats as a liveness check and nothing more: a saturated OSD passes them while delaying every write it touches, so latency monitoring through ceph osd perf is what actually detects it. Open the full OSD port range on both networks rather than the ports you observed carrying traffic, since heartbeats use dedicated addresses. And when OSDs flap, work the interface counters, latency, and host load before considering heartbeat timing — widening the grace window declines to notice the fault and delays every real failure by the same amount.

Cross-course references

Ceph: Part LV (OSD States) for down and out in detail.
Ceph: Part XXX (Network Failure Behaviour) for the network side.
Ceph: Part LXVIII (OSD Latency) for detecting the slow-but-alive case.