Skip to main content
RunBook Academy

CephXXX · Network Failure BehaviourNetwork Failure Behaviour

How network latency propagates into storage latency

Intermediate⏱ ~16 minpingceph

What you'll learn

  • Compute the network contribution to write latency
  • Explain why replication multiplies the RTT contribution
  • Measure the network term separately from the storage term
  • Set realistic expectations for stretched deployments

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Not yet marked complete on this device.

Why this matters in production

Network latency is not a small additive term for Ceph — it appears twice in the write path and is paid on every single operation. On all-flash clusters it is frequently the dominant component, which surprises teams who spent their budget on the drives.

The write path arithmetic

client → primary               : RTT/2
primary → replicas             : RTT/2
replica device commit          : device latency
replicas → primary             : RTT/2
primary → client               : RTT/2

Total network contribution: 2 × RTT. Plus the slowest replica’s device commit.

Network RTTDeviceNetwork share of total
0.1 msNVMe, 0.1 ms0.2 / 0.3 = 67%
0.1 msHDD, 8 ms0.2 / 8.2 = 2%
2 msNVMe, 0.1 ms4 / 4.1 = 98%
2 msHDD, 8 ms4 / 12 = 33%

Row three is the trap: an all-NVMe cluster on a 2 ms network delivers worse write latency than an HDD cluster on a fast one, and every euro spent on flash is wasted.

Measuring the network term

# baseline RTT between client and OSD hosts
ping -c 1000 -i 0.01 10.20.0.47 | tail -2

# and between OSD hosts, for the replication hop
ping -c 1000 -i 0.01 10.30.0.83 | tail -2

Then compare against observed operation latency:

ceph osd pool stats rbd-vms
ceph daemon osd.12 dump_historic_ops

If 2 × RTT accounts for most of the observed write latency, tuning storage will not help.

Stretched clusters

Ceph across sites pays the inter-site RTT on every write:

SeparationTypical RTTAdded write latency
Same rack0.05 ms0.1 ms
Same datacentre0.2 ms0.4 ms
Metro, 30 km0.4 ms0.8 ms
Regional, 300 km4 ms8 ms
Continental40 ms+80 ms+

Metro distances are workable for many workloads. Regional and beyond are not, for synchronous replication — which is why cross-region Ceph uses asynchronous approaches such as RBD mirroring or RGW multisite rather than stretching a single cluster.

Quiz

Knowledge check · 4 questions

  1. Q1. A cluster has 0.5 ms network RTT and NVMe devices with 0.1 ms commit latency. What dominates write latency?

  2. Q2. Reads pay one network round trip while writes pay two.

  3. Q3. Advise on an all-flash upgrade.

    A team proposes replacing HDD OSDs with NVMe to reduce RBD write latency from a measured 11 ms. The cluster spans two buildings with a measured 1.8 ms RTT between them, and the CRUSH failure domain is set so replicas always span both buildings.

  4. Q4. Why is a read/write latency ratio far from the devices' own characteristics a network signal?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Measure and record client-to-OSD and OSD-to-OSD RTT as part of cluster commissioning, and re-check after any topology change. Include the network floor in any latency improvement proposal so expectations match what the physical layout permits — it is a cheap calculation and prevents expensive disappointment.

Cross-course references

  • Kubernetes: cross-zone traffic adds the same per-operation latency to distributed workloads
  • Linux: synchronous replication over distance has this well-known and unavoidable floor