Choosing link speed for a Ceph cluster

Intermediate⏱ ~16 mincephiperf3

What you'll learn

Convert link speed to realistic usable throughput
Derive the network requirement from OSD count and device speed
Match link speed to recovery-window requirements
Justify a speed choice with arithmetic

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Link speed is a long-lived decision — you will not re-cable a cluster casually — and the common advice (“10 Gb minimum”) is a floor rather than an answer. The right figure comes from the devices in the chassis and the recovery window you are willing to accept.

Usable throughput

Nominal	Theoretical	Realistic sustained
10 GbE	1.25 GB/s	~1.1 GB/s
25 GbE	3.1 GB/s	~2.8 GB/s
40 GbE	5.0 GB/s	~4.4 GB/s
100 GbE	12.5 GB/s	~11 GB/s

Plan against the third column, and remember that client and replication traffic share it unless you have a separate cluster network.

Deriving the requirement

From the devices. Sum what the OSDs in one host can actually deliver:

12 × HDD at 180 MB/s sequential   = 2.2 GB/s   → 25 GbE
12 × SATA SSD at 500 MB/s         = 6.0 GB/s   → 100 GbE
8 × NVMe at 3 GB/s                = 24 GB/s    → 100 GbE, still short

A host whose devices can saturate the link means the network, not the storage, sets throughput. For HDD hosts, 25 GbE is generally ample; for NVMe, the network is the constraint at any commercially reasonable speed and the question becomes how much of the device capability you are willing to leave unused.

From the recovery window. This is usually the binding requirement:

one 16 TB OSD, EC 8+3 → 128 TB of recovery traffic
at 50% of a 25 GbE link (1.4 GB/s) → 25 hours
at 50% of a 100 GbE link (5.5 GB/s) → 6.5 hours

Compare against the degraded window you will accept, and let that comparison choose the speed.

Practical guidance

Cluster	Public	Cluster
Small HDD, under 10 hosts	10 GbE	shared
Production HDD	25 GbE	25 GbE
SATA SSD	25 GbE	25 GbE or 100
NVMe	100 GbE	100 GbE
Wide EC on large drives	25 GbE	100 GbE

The last row is worth noting: EC recovery volume can justify a faster cluster network than the client-facing side needs.

Verify what you actually have

ethtool ens1f0 | grep -E 'Speed|Duplex'
iperf3 -c 10.30.0.47 -P 8 -t 30

Run iperf3 between OSD hosts on both networks during commissioning and record the result. A 25 GbE link negotiating at 10 GbE, or delivering 6 Gb/s because of a host-side bottleneck, is far easier to find at build time than during an incident.

Why NVMe clusters are network-bound at any practical speed

A single modern NVMe device sustains several gigabytes per second, so a host with eight of them can deliver more than twice what a 100 GbE link carries. Ceph’s replication multiplier makes this worse: a size-3 write consumes device bandwidth on three OSDs and network bandwidth twice over. The practical consequence is that all-NVMe Ceph clusters are provisioned knowing the devices will not run at their rated throughput, and the design question becomes how much aggregate IOPS and latency the deployment needs rather than how to saturate the drives. This is also why NVMe clusters benefit disproportionately from RDMA-capable fabrics and from careful attention to host-side CPU, which become the next constraints once the link is the bottleneck.

Quiz

Knowledge check · 4 questions

Q1. A host has 12 HDD OSDs each capable of about 180 MB/s sequential. What link speed matches that capability?

10 GbE, whose 1.1 GB/s covers HDD sequential work 100 GbE, since replication triples the host traffic 25 GbE, since the devices can deliver about 2.2 GB/s Link speed is irrelevant when the OSDs are all HDD
Q2. For most new builds, 25 GbE is a better choice than 40 GbE.

True False
Q3. Choose link speed from a recovery requirement.

A design uses 18 TB drives with EC 8+3 across 16 hosts. The operations team requires return to full redundancy within 24 hours of a single OSD failure. Both 25 GbE and 100 GbE are within budget, with 100 GbE costing significantly more.
Q4. Why should iperf3 be run between OSD hosts during commissioning?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Derive link speed from the recovery-window requirement and the per-host device capability, and show both calculations in the design. Measure achieved throughput with iperf3 at commissioning and store the result alongside the design document — the gap between rated and achieved is exactly the sort of thing that is impossible to reconstruct later.

Cross-course references

Kubernetes: node network capacity planning for storage-backed workloads uses the same derivation
Linux: verifying negotiated link speed with ethtool is standard practice for the same reason