Which network partitioned, and how the symptoms differ

Advanced⏱ ~18 mincephss

What you'll learn

Map each Ceph traffic class to the network that carries it
Read front and back heartbeat failures from logs and health checks
Distinguish a public-network split from a cluster-network split by symptom
Verify the addresses actually in use before testing a path

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

A cluster with a separate cluster network has two independent things that can partition, and they present differently enough that the symptom alone tells you which cable to chase. Getting it wrong sends the network team to the correct rack and the wrong switch.

What rides on each network

Traffic	Public	Cluster
Client to monitor	yes	no
Client to OSD	yes	no
OSD to monitor	yes	no
OSD to OSD replication and recovery	no	yes
OSD heartbeat, front	yes	no
OSD heartbeat, back	no	yes
MGR, MDS, RGW	yes	no

ceph config get osd public_network
ceph config get osd cluster_network

If cluster_network is unset, everything is on the public network and there is only one thing to partition — which is a simpler cluster to reason about and a defensible design choice.

Reading front and back from the evidence

ceph health detail
ceph daemon osd.7 dump_osd_network 0
grep heartbeat_check /var/log/ceph/$FSID/ceph-osd.7.log | tail -5

osd.7 heartbeat_check: no reply from 10.30.4.11:6803 osd.42 ever on either
  front or back, first ping sent 2026-08-18T09:13:38 (oldest deadline
  2026-08-18T09:13:58)

osd.7 heartbeat_check: no reply from 10.30.4.11:6803 osd.42 since back
  2026-08-18T09:13:38 front 2026-08-18T09:14:11

The wording is the diagnosis. either front or back means both paths are gone. A message naming only back, with a recent front timestamp, means the public path is fine and the cluster network is not. The health checks OSD_SLOW_PING_TIME_FRONT and OSD_SLOW_PING_TIME_BACK make the same distinction for degradation short of total loss.

The two signatures

Symptom	Public split	Cluster split
Client connects	fails or times out	succeeds
Client reads	fail on the isolated side	succeed from the primary
Client writes	fail on the isolated side	connect, then hang
Monitors reachable from far OSDs	no	yes
OSDs marked down	yes	yes
Marked-down OSDs contest it	no	yes, loudly
Dashboard and MGR view	blind to the far side	complete

A cluster-network split is the more confusing of the two precisely because everything answers. Clients connect, monitors are healthy, the dashboard is fully populated — and writes hang, because a primary cannot replicate to an acting set member it can no longer reach.

Verifying the path before you test it

ceph osd metadata 42 | grep -E 'front_addr|back_addr|hb_front_addr|hb_back_addr'

# then test that exact address and port from the peer, in both directions
ss -tn state established '( dport = :6802 or dport = :6803 )'

Testing the address you assume an OSD uses, rather than the one it reports, is how a cluster network with a mis-set subnet survives an entire diagnostic session undetected.

Quiz

Knowledge check · 4 questions

Q1. Clients connect normally and monitors are quorate, but writes to certain pools hang. Which split does this fit?

A monitor-only partition, which leaves clients connected to their OSDs while every map update stalls behind the lost quorum A cluster-network split — the public path is intact, so everything answers, while replication across the break cannot complete A public-network split confined to the client subnets, so established sessions survive while new writes cannot be routed at all A CRUSH map error rather than a network fault, since a rule that can no longer place a PG blocks every write to that pool
Q2. An OSD can be correctly marked down while it is fully reachable on the public network and answering commands from its own host.

True False
Q3. Localise a partition to one of two networks.

Eight OSDs across two hosts are down. The hosts respond to SSH, `ceph -s` runs fine from them, and their OSD logs show heartbeat_check messages naming only back.
Q4. What does a heartbeat_check message naming only back, with a recent front timestamp, tell you?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Read the heartbeat message wording before testing anything — front, back, or both is the fastest localisation you will get, and it costs one grep. Take the addresses to test from ceph osd metadata rather than from the configuration you believe is in force, because a cluster network bound to an unexpected subnet passes every test aimed at the expected one.

Cross-course references

Kubernetes: control-plane reachability and data-plane reachability fail independently and present differently
Linux: a bonded interface can pass one traffic class and drop another for hours