Slow ops originating in the network

Advanced⏱ ~17 mincephiperf3ethtool

What you'll learn

Recognise network signatures in slow ops
Test the network path from the implicated hosts
Distinguish loss, congestion, and misconfiguration
Apply the appropriate remedy

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Network problems produce slow ops that look like OSD problems, and the distinguishing evidence is at the host and interface level rather than in Ceph.

The signatures

Signature	Network cause
Many OSDs report slow ops with no device correlation	loss or congestion
`waiting for subops` between specific host pairs	a path between those hosts
Latency clustering near 200 ms	TCP retransmission timeout
OSDs flapping alongside slow ops	heartbeat loss
Slow ops appear only under load	congestion, buffer exhaustion
Slow ops correlate with recovery	bandwidth saturation
Large transfers fail, small ones work	MTU mismatch

# how many OSDs and on how many hosts?
ceph health detail | grep -oE 'osd\.[0-9]+' | sort -u | wc -l
ceph health detail | grep -oE 'osd\.[0-9]+' | sort -u | while read o; do
  ceph osd find "${o#osd.}" --format json 2>/dev/null | \
    python3 -c 'import sys,json; print(json.load(sys.stdin)["host"])'
done | sort | uniq -c

Slow ops spread across many hosts with no device pattern points at the fabric.

Testing the path

# between the implicated hosts, both directions
PEER=peer
iperf3 -s                          # on one
iperf3 -c ${PEER} -t 30 -P 8        # forward
iperf3 -c ${PEER} -t 30 -P 8 -R     # reverse

# latency under load
ping -c 1000 -i 0.01 ${PEER} &
iperf3 -c ${PEER} -t 15 -P 8
wait

# interface evidence
IFACE=bond0
ethtool -S ${IFACE} | grep -iE 'drop|discard|err|pause|no_buffer'
ip -s link show ${IFACE}
netstat -s | grep -iE 'retransmit|timeout'

Distinguishing the causes

Evidence	Cause
Interface error counters rising	a physical problem — cable, transceiver, port
Drops with no errors	buffer exhaustion, congestion
Retransmits without interface errors	loss elsewhere in the path
Throughput far below link speed	duplex, MTU, or a degraded path
Latency fine idle, poor under load	congestion
`ping -M do` fails at full MTU	MTU mismatch
Bond member count below configured	reduced capacity

# MTU across every pair
for h in $(ceph orch host ls --format json | python3 -c \
  'import sys,json;[print(x["addr"]) for x in json.load(sys.stdin)]'); do
  printf '%-16s ' "$h"
  ping -M do -s 8972 -c 2 -W 2 "$h" >/dev/null 2>&1 && echo OK || echo FAIL
done

Remedies

Cause	Remedy
Physical fault	replace the cable or transceiver
Congestion	more capacity, or reduce recovery concurrency
MTU mismatch	correct the mismatched element
Degraded bond	restore the member
Buffer exhaustion	deeper buffers, PFC, or lower burst

# immediate mitigation while the fabric is fixed
ceph config set osd osd_max_backfills 1
ceph config set osd osd_recovery_max_active_hdd 1

Quiz

Knowledge check · 4 questions

Q1. Why do OSD flapping and slow ops appearing together point specifically at the network?

Flapping always produces slow ops, because every transition repeers the OSD's PGs and holds their operations Packet loss affects both heartbeats and data operations, while a device fault produces slow ops without causing peers to mark the OSD down The monitors correlate the two events and raise the slow-op warning whenever an OSD has recently been marked down Flapping triggers recovery on every transition, and it is that recovery load rather than the network which ages the operations
Q2. A forward `iperf3` test between the implicated hosts is sufficient to exonerate the network.

True False
Q3. Investigate slow ops with no device pattern.

Slow ops are reported on 14 OSDs spread across 9 different hosts. Device models and ages vary. SMART is clean on all of them. Some OSDs have also flapped.
Q4. What distinguishes interface drops from interface errors, and what does each indicate?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Treat slow ops scattered across many hosts with no device pattern as a fabric signature and stop investigating devices. Test both directions under load — a forward idle-link test proves almost nothing — and read flapping alongside slow ops as strong network evidence.

Cross-course references

Kubernetes: errors spread across nodes with no node pattern point at the underlay
Linux: correlated symptoms across independent hardware implicate the shared layer