One-way reachability and the flap it produces

Advanced⏱ ~18 mincephssping

What you'll learn

Trace the flap loop from heartbeat loss to map re-peering
List the faults that produce one-way reachability
Test each direction independently rather than testing connectivity
Stop a flap safely while the fault is diagnosed

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

An OSD that goes down, comes back, and goes down again is almost never a failing disk. It is a directional network fault, and the flapping it causes damages client latency far more than the underlying fault would if the OSD simply stayed down.

The flap loop

1. Peers stop receiving from osd.42 and file failure reports
2. The monitors commit a new OSDMap with osd.42 down
3. osd.42 reads that map — its monitor session is unaffected — sees itself
   marked down, and objects
4. The monitors commit another map with osd.42 up
5. Every PG on osd.42 re-peers, twice
6. Twenty seconds later, step 1

2026-08-18T09:14:22.884 osd.42 map e91847 wrongly marked me down at e91846

That message is the diagnosis. An OSD can only send it if it reached a monitor, which means the public path to the monitors is intact while the heartbeat path to its peers is not.

Where the asymmetry comes from

Fault	Why it is directional
Asymmetric routing	return path traverses a different, broken link
Stateful firewall rule applied on one host	inbound blocked, outbound permitted
Conntrack table exhaustion	new flows dropped in one direction only
Bond member failing on transmit	receive still works on the surviving member
Path MTU blackhole	large frames dropped one way, small ones pass
ACL or security group added to one side	the change was never mirrored

Testing direction, not connectivity

# from osd.42's host, toward a peer
ping -c3 10.20.4.11
ss -tn state established '( dport = :6802 or sport = :6802 )'

# then the same two commands from the peer, toward osd.42
# a successful ping in one direction proves nothing about the other
ceph daemon osd.7 dump_osd_network 0

dump_osd_network reports measured ping times per peer for front and back interfaces, so a peer that appears in one direction’s table and not the other localises the break to a path rather than a host.

Stopping the flap while you diagnose

ceph osd set nodown

# and afterwards, without fail
ceph osd unset nodown

Flag	Effect	Cost while set
`nodown`	monitors ignore failure reports	a genuinely dead OSD stays up in the map
`noup`	OSDs cannot rejoin	prevents a mass re-peer storm on repair
`noout`	down OSDs are not removed by CRUSH	no automatic rebuild if the loss is real

nodown freezes the map so peering stops churning and you get a stable cluster to investigate. It is a diagnostic aid with a real cost, so it belongs in the incident log with an owner and an expected removal time.

Quiz

Knowledge check · 4 questions

Q1. An OSD logs that it was wrongly marked down and rejoins, repeatedly. What does the message prove about the fault?

The public path to the monitors is intact while the peer heartbeat path is not, so the fault is directional The OSD process is restarting in a crash loop, and each new instance re-registers with the monitors before failing The monitors are losing quorum intermittently, so the committed map alternates between two conflicting views The disk beneath the OSD is returning read errors, which the OSD reports to its peers as a heartbeat timeout
Q2. A flap that stops after several cycles indicates the underlying network fault has cleared.

True False
Q3. Contain a flapping OSD during a live incident.

osd.42 has cycled down and up eleven times in nine minutes. Client write latency on several pools has risen sharply. A ping from the OSD host to its peers succeeds.
Q4. Why does flapping harm clients more than the same OSD staying cleanly down?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Treat repeated wrongly marked me down messages as a network finding, not an OSD finding, and always test the reverse direction from the peer host before concluding anything. Set nodown early to stop the peering churn, and log the flag with an owner and a removal time — an unnoticed nodown means the next genuine failure goes undetected.

Cross-course references

Kubernetes: a pod that passes readiness locally but fails from the service mesh has the same directional shape
Linux: a successful ping proves one direction of one protocol at one packet size