The network component of RBD latency

Intermediate⏱ ~16 minpingiperf3ceph

What you'll learn

Compute the network contribution to RBD operation latency
Measure RTT along both the client and replication paths
Determine when the network is the dominant term
Reduce the network contribution

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

On flash clusters the network is usually the largest single component of write latency, and it is invisible in every storage metric. Measuring it separately is what prevents a performance project from optimising the part that is already fast.

The two round trips

client → primary        : RTT_client / 2
primary → replicas      : RTT_cluster / 2
replicas commit         : device latency
replicas → primary      : RTT_cluster / 2
primary → client        : RTT_client / 2

total network = RTT_client + RTT_cluster

Reads pay only RTT_client, which is why read and write latency diverge as the network slows.

When it dominates

Device	Device latency	Network at 0.1 ms RTT	Network share
HDD	8 ms	0.2 ms	2%
SATA SSD	0.5 ms	0.2 ms	29%
NVMe	0.08 ms	0.2 ms	71%

On NVMe the network is the majority of the operation. Buying faster devices without addressing the network buys very little.

Measuring

# client to OSD hosts
ping -c 1000 -i 0.01 10.20.0.47 | tail -2

# OSD host to OSD host, the replication path
ping -c 1000 -i 0.01 10.30.0.83 | tail -2

# under load, which is what matters
ping -c 1000 -i 0.01 -f 10.30.0.83   # flood, as root

Measure under load. An idle-network RTT understates what operations actually experience when the links are carrying traffic.

Compare against observed operation latency:

ceph daemon osd.12 dump_historic_ops | \
  jq -r '.ops[0].type_data.events[] | "\(.time) \(.event)"'

The gaps between events localise where the time goes — and the gap between dispatch to replicas and their acknowledgement is the cluster-network term.

Reducing it

Action	Effect
Fewer switch hops	direct RTT reduction
Cut-through rather than store-and-forward switching	reduces per-hop latency
Avoid routing where a flat L2 segment would do	removes router latency
Co-locate clients with OSDs where topology permits	shortest path
Jumbo frames	marginal for small I/O
Faster link speed	helps throughput, little for latency

The most effective changes are topological. Link speed addresses bandwidth, not latency, and is frequently the change proposed when latency is the problem.

Quiz

Knowledge check · 4 questions

Q1. A cluster has NVMe OSDs with 0.08 ms device latency and 0.3 ms network RTT on both paths. What dominates write latency?

The devices, because the NVMe commit still bounds when the write completes CRUSH computation, which the client repeats for every object it writes The monitors, which must acknowledge each write before the primary replies The network, contributing roughly 0.6 ms against 0.08 ms of device time
Q2. Upgrading from 10 GbE to 100 GbE substantially reduces the latency of a single 4 KiB write.

True False
Q3. Sequence a latency improvement project.

A team wants to reduce RBD write latency from 4.2 ms. The cluster uses SATA SSDs with about 0.5 ms commit latency. Measured RTT is 1.6 ms on the client path and 1.5 ms on the cluster path, both traversing four switch hops through a routed fabric.
Q4. Why does write latency diverge from read latency as a network slows?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Measure RTT on both the client and cluster paths under load and record it alongside device latency, so any latency proposal can be evaluated against the actual breakdown. Sequence network work ahead of device purchases when the network term dominates — the arithmetic is quick and it frequently redirects a project.

Cross-course references

Kubernetes: cross-zone latency dominates distributed storage operations the same way
Linux: RTT versus device service time is the standard decomposition for any networked storage