Packet loss and why it destabilises OSDs

Why this matters in production

A 0.1% packet loss rate sounds negligible and is catastrophic for Ceph. It is invisible to a casual ping, it does not trigger any network alarm, and it produces symptoms — flapping OSDs, slow ops, stalled recovery — that look exactly like storage problems. Recognising the signature saves days.

Why small loss rates matter so much

Ceph’s traffic is many small request/response exchanges over TCP. Every lost packet costs a retransmission timeout, and TCP’s minimum RTO is typically 200 ms — three orders of magnitude above a healthy sub-millisecond Ceph operation.

0.1% loss on operations of 10 packets each
  → ~1% of operations lose a packet
  → those operations take 200 ms instead of 0.5 ms
  → p99 latency jumps by 400×

The mean barely moves. The tail is destroyed. And because Ceph writes wait for all replicas, one loss anywhere in the acting set delays the whole operation.

The OSD flapping mechanism

Heartbeats are small and frequent. Losing enough of them within osd_heartbeat_grace (20 s by default) causes peers to report an OSD down:

1. Loss delays or drops heartbeats
2. Peers report the OSD down; monitors mark it down
3. The OSD, alive and well, sees itself marked down and reports back up
4. PGs re-peer, recovery starts, then aborts
5. Repeat

ceph -w | grep -E 'osd\.[0-9]+ (down|up)'
journalctl -u ceph-osd@12 | grep -i 'wrongly marked me down'

wrongly marked me down is the definitive log line. An OSD saying it was wrongly marked down is telling you its network path failed, not its disk.

Measuring properly

# sustained, high count — 10 pings prove nothing
ping -c 10000 -i 0.01 10.30.0.47 | tail -3

# per-hop loss, which localises it
mtr --report --report-cycles 1000 10.30.0.47

# host-side TCP counters
ss -s
nstat -az | grep -iE 'retrans|drop|fail'
netstat -s | grep -iE 'retransmit|bad segment'

TcpRetransSegs climbing steadily relative to TcpOutSegs is loss. Interface-level rx_dropped/tx_dropped in ip -s link point at the host; growth on switch port counters points at the fabric.

Loss versus congestion versus fault

The last row matters: an MTU problem presents as loss in casual testing and needs a completely different fix.

Quiz

Production discipline

Treat OSD flapping without device errors as a network incident from the first minute, and set noout early so the cluster stops moving data in response to a fault that is not storage-related. Monitor TCP retransmission rates on OSD hosts continuously — they detect the condition long before flapping starts.

Cross-course references

• Kubernetes: pod readiness flapping from network loss follows the identical pattern
• Linux: TCP retransmission counters in nstat are the standard first measurement