The network the design assumes

Advanced⏱ ~18 mincephiperf3ping

What you'll learn

Size the network from drive throughput rather than client demand
Split public and cluster traffic where it pays
Configure a bond whose hash actually spreads OSD traffic
Recognise the failure modes MTU and network splits introduce

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Ceph fails softly when the network is marginal — slow ops, flapping OSDs, recoveries that never finish — and each of those looks like a storage problem to everyone except the person holding the switch counters.

What each network carries

Network	Traffic	Volume per 1 GiB of client write
Public	client IO, mon and mgr sessions	1 GiB in
Cluster	replication at size=3	2 GiB
Cluster	erasure coding at 4+2	1.25 GiB
Cluster	backfill, recovery, scrub reads, OSD heartbeats	unbounded by client rate

ceph config set global public_network 10.20.0.0/24
ceph config set global cluster_network 10.30.0.0/24
ceph config get osd cluster_network

Both settings are read at daemon start. Changing them on a running cluster requires restarting the OSDs, one failure domain at a time.

Sizing from the drives

12 HDDs x 190 MB/s sequential = 2.3 GB/s = about 18 Gbit/s per node.
That is the rate a node can absorb during backfill, and the reason
10 GbE is the bottleneck on an HDD node rather than a comfortable fit.

Node type	Aggregate drive throughput	Network
12 HDD	~18 Gbit/s	2 x 25 GbE
24 SATA SSD	~100 Gbit/s	2 x 100 GbE
10 NVMe	~200 Gbit/s	2 x 100 GbE, and the CPU binds first

Rebuilding one 216 TB host across a 20 Gbit/s path is 24 hours of wire time. The defaults will not deliver that: the mclock scheduler deliberately protects client IO.

ceph config set osd osd_mclock_profile high_recovery_ops
ceph config get osd osd_mclock_profile
# return to the default once the rebuild completes
ceph config set osd osd_mclock_profile balanced

The bond

# /etc/netplan/60-ceph.yaml
network:
  bonds:
    bond0:
      interfaces: [ens1f0, ens1f1]
      mtu: 9000
      parameters:
        mode: 802.3ad
        lacp-rate: fast
        mii-monitor-interval: 100
        transmit-hash-policy: layer3+4
  vlans:
    bond0.20: {id: 20, link: bond0, mtu: 9000, addresses: [10.20.0.11/24]}
    bond0.30: {id: 30, link: bond0, mtu: 9000, addresses: [10.30.0.11/24]}

cat /sys/class/net/bond0/bonding/xmit_hash_policy
cat /proc/net/bonding/bond0 | grep -E 'MII Status|Aggregator ID'
iperf3 -c 10.30.0.12 -P 8 -t 30

If eight parallel streams cap at 25 Gbit/s on a 2 x 25 bond, the hash policy is wrong on one side. Both the host and the switch must hash on layer3+4.

MTU

# 9000 minus 28 bytes of IP and ICMP header, with fragmentation forbidden
ping -M do -s 8972 -c 3 10.30.0.12
ip -br link show bond0.30

Symptom	Cause
Small IO fine, large IO hangs	MTU mismatch somewhere on the path
Monitors healthy, OSDs flapping	the mismatch is on the cluster VLAN only
Works host to host, fails through a router	an intermediate hop at 1500

Set 9000 on the bond, on every VLAN interface, and on every switch port including the inter-switch links. One port left at 1500 produces intermittent failures that survive every reboot.

Why LACP with a layer2 hash puts all OSD traffic on one link

An LACP bond does not stripe packets; it selects one member link per flow so that packets never reorder. The selection is a hash. With the default layer2 policy the hash is over source and destination MAC addresses, and between two Ceph nodes those are constant — so every OSD-to-OSD flow between that pair lands on the same physical link, and a 2 x 25 GbE bond delivers 25 Gbit/s between any two hosts no matter how many OSDs are talking. The layer3+4 policy hashes source and destination IP and port, and since each OSD binds its own ports the flows spread across members. The consequence is that a bond can double aggregate cluster bandwidth while doing nothing at all for the pairwise rebuild rate that determines how long a failed host takes to recover.

Quiz

Knowledge check · 4 questions

Q1. A 2 x 25 GbE LACP bond delivers only 25 Gbit/s between two OSD nodes even with many parallel flows. What is the most likely cause?

The switch fabric is oversubscribed, so the pair of hosts shares uplink capacity with everything else in the rack The transmit hash policy is layer2, so every flow between the same pair of MAC addresses selects the same member link Jumbo frames are disabled on one of the VLAN interfaces, so every frame is fragmented and half the throughput is header The cluster network is saturated by scrub reads, which are unbounded by client rate and consume whatever is left
Q2. Adding a separate cluster network reduces the number of ways the cluster can lose OSDs.

True False
Q3. Diagnose intermittent OSD flapping on a new cluster.

A freshly built six-node cluster reports OSDs going down and coming back several times an hour. Small IO works, large sequential writes stall. The monitors have never lost quorum.
Q4. What sets the required network speed for a storage node?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Size the network from what the drives can move during a rebuild, not from what clients ask for at steady state — recovery is the case that decides whether the cluster returns to full redundancy in a day or a week. Verify MTU with a do-not-fragment ping across every pair before the cluster carries data, and verify the LACP hash with parallel streams rather than a single one.

Cross-course references

Kubernetes: CNI MTU mismatches produce the same small-works, large-hangs signature
Linux: bonding never splits a single flow, so aggregate bandwidth and pairwise bandwidth are different numbers