When storage latency is really network latency

Advanced⏱ ~17 miniperf3pingceph

What you'll learn

Count the network round trips in a Ceph operation
Estimate the network's share of observed latency
Test the network in isolation
Recognise the network signatures

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Every Ceph write crosses the network at least twice and often four times. A network adding two milliseconds per hop adds eight milliseconds to a write, which looks exactly like slow storage.

Counting the round trips

sequenceDiagram
  participant C as Client
  participant P as Primary OSD
  participant R as Replicas
  C->>P: write (public network)
  P->>R: subop (cluster network)
  R-->>P: subop ack (cluster network)
  P-->>C: ack (public network)

Hop	Network
Client → primary	public
Primary → replicas	cluster
Replicas → primary	cluster
Primary → client	public

Two full round trips minimum. Erasure coding adds more, since k+m OSDs must be contacted rather than size.

OSD_HOST=stor-04
ping -c 1000 -i 0.01 ${OSD_HOST} | tail -2

rtt min/avg/max/mdev = 0.081/0.112/1.204/0.043 ms

network contribution ≈ 2 × 0.112 ms ≈ 0.22 ms

Against a 12 ms HDD write that is negligible. Against a 200 µs NVMe write it is more than the device — which is why all-flash clusters are far more sensitive to network latency, and why 25 GbE with low-latency switching matters there and not on HDD.

Cluster	Device latency	Network share of total
HDD	10 ms	~2%
SATA SSD	0.3 ms	~40%
NVMe	0.08 ms	~70%

Testing in isolation

# throughput
HOST=stor-04
iperf3 -s                              # on one OSD host
iperf3 -c ${HOST} -t 30 -P 4            # on another

# latency under load
ping -c 1000 -i 0.01 ${HOST} &
iperf3 -c ${HOST} -t 15 -P 8
wait

Latency measured while the link is loaded is the number that matters — an idle-link ping proves very little about behaviour under Ceph traffic.

# per-interface counters
IFACE=bond0
ip -s link show ${IFACE}
ethtool -S ${IFACE} | grep -iE 'drop|discard|err|pause'

The network signatures

Signature	Cause
Latency spikes with no device correlation	congestion or buffering
Retransmits in `netstat -s`	loss
Throughput far below link speed	duplex, MTU, or a bad path
Latency rises only under load	switch buffer exhaustion
One host slow to all others	that host’s NIC or cabling
All hosts slow to one host	that host’s NIC or its switch port

netstat -s | grep -iE 'retransmit|segments retransmited'
ss -ti | grep -A1 'cwnd' | head -20

Quiz

Knowledge check · 4 questions

Q1. Why do all-flash clusters suffer more from network latency than HDD clusters?

The device contributes microseconds rather than milliseconds, so two round trips can exceed the device time Flash-backed pools need more round trips per write, because acknowledgement is split across commit and apply Flash clusters are configured with larger objects, so each write moves more bytes across the network HDD clusters compress their replication traffic, which reduces what crosses the cluster network per write
Q2. An idle-link ping is a valid measurement of the latency Ceph will experience.

True False
Q3. Investigate latency after an all-flash refresh.

A cluster was refreshed from HDD to NVMe. Device latency fell from 11 ms to 0.09 ms but client-observed latency only fell from 14 ms to 4 ms. The network was not changed.
Q4. How many network round trips does a size=3 replicated write require, and where?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Measure network latency under load rather than on an idle link; switch buffering is what adds the latency Ceph experiences and the idle figure can be an order of magnitude lower. Re-examine network specifications after any all-flash refresh — the network that was irrelevant on HDD becomes the dominant term.

Cross-course references

Kubernetes: service mesh latency becomes dominant as backend latency falls
Linux: network filesystem latency is dominated by RTT when the server is fast