Network design for a dedicated cluster

Why this matters in production

The network is the one component that cannot be upgraded incrementally without disruption, and it is sized on assumptions that are easy to get wrong.

Sizing

public network:  peak aggregate client I/O
cluster network: public write traffic × (size - 1) + recovery traffic

Client workload: 4 GB/s peak, 70% read / 30% write
  public:  4 GB/s = 32 Gb/s
  writes:  1.2 GB/s
  cluster: 1.2 × 2 = 2.4 GB/s for replication
         + recovery, which can be several GB/s
  → cluster network should exceed the public network

# measure actual usage
ceph -s | grep client
sar -n DEV 1 30 | awk '/ens|bond/ {rx+=$5; tx+=$6} END {print rx/30, tx/30, "kB/s"}'

The recovery term is what most sizing exercises omit, and it is the term that matters during the events the cluster exists to survive.

Configuration

ceph config set global public_network 10.0.2.0/24
ceph config set global cluster_network 10.0.3.0/24
ceph config get osd public_network

# verify OSDs bound to both
ceph osd metadata 0 | python3 -c '
import sys,json; d=json.load(sys.stdin)
print("front:", d.get("front_addr"))
print("back: ", d.get("back_addr"))'

A cluster network configured but with OSDs not bound to it is a common misconfiguration that is silent — the setting is present and unused.

Link redundancy

# bond configuration
cat /proc/net/bonding/bond0 | grep -E 'Mode|Hash|MII Status'

# alert on a degraded bond
cat /proc/net/bonding/bond0 | grep -c 'MII Status: up'

Verifying the design

# throughput, both directions, multiple streams
PEER=peer
iperf3 -c ${PEER} -t 30 -P 8
iperf3 -c ${PEER} -t 30 -P 8 -R

# latency under load
ping -c 1000 -i 0.01 ${PEER} &
iperf3 -c ${PEER} -t 15 -P 8
wait

# MTU across every pair
for h in $(ceph orch host ls --format json | python3 -c \
  'import sys,json;[print(x["addr"]) for x in json.load(sys.stdin)]'); do
  printf '%-16s ' "$h"
  ping -M do -s 8972 -c 2 -W 2 "$h" >/dev/null 2>&1 && echo OK || echo FAIL
done

# the real test: a full recovery
OSD=12
ceph osd out ${OSD}
# watch network utilisation and client latency throughout

Planning for growth

Quiz

Production discipline

Size the cluster network above the public network — it carries size - 1 copies of every write plus all recovery traffic. Verify OSDs are actually bound to the cluster network with ceph osd metadata; a setting added without restarting them is silently unused.

Cross-course references

• Kubernetes: control and data plane network sizing follows the same asymmetry
• Linux: replication traffic exceeding client traffic is normal in any replicated store