Network design for a production cluster

Why this matters in production

Networks are decided at build time and are expensive to change. Getting the count and the sizing right means understanding what each carries, which is a short list.

The networks

Ceph itself uses the first two. A management network is an operational choice: it keeps administrative access working when the storage networks are saturated, which is precisely when you need it.

ceph config set global public_network 10.20.0.0/16
ceph config set global cluster_network 10.30.0.0/16

There is no separate network for credentials or authentication — cephx traffic rides the public network with everything else.

Sizing

Public network — sized against client demand, remembering that RGW and MDS traffic to OSDs also rides it:

client read demand + client write demand + RGW-to-OSD + MDS-to-OSD

Cluster network — sized against replication and recovery:

replication:  (size − 1) × client write rate
recovery:     bounded only by throttles; the peak driver

Recovery is what sizes the cluster network. Compute the traffic for rebuilding your largest OSD and divide by the acceptable window.

Without a cluster network, everything shares the public link and the sum is what matters:

size-3 pool, 1 GB/s of client writes
  → 1 GB/s client + 2 GB/s replication = 3 GB/s on one network

Whether to separate

Separate when: recovery demonstrably affects client latency; the cluster is large; EC multiplies recovery volume; two fabrics are operationally manageable.

Do not separate when: a single faster network has ample headroom; the team cannot troubleshoot two paths; the cluster is small.

A single well-provisioned 25 or 100 Gb network frequently beats two congested 10 Gb ones, and it has one failure mode instead of three.

Verifying before deployment

# between every pair of storage hosts, on every network
iperf3 -c 10.30.0.21 -P 8 -t 30

# MTU, if jumbo frames are in use
ping -M do -s 8972 -c 3 10.30.0.21

# latency baseline
ping -c 1000 -i 0.01 10.30.0.21 | tail -2

Record the results. They are the baseline every future investigation compares against, and they are impossible to reconstruct later.

Quiz

Production discipline

Size the cluster network against recovery rather than replication; replication is bounded by client demand and recovery is not. Record iperf3, MTU, and latency baselines at commissioning — they are the reference every later investigation needs and cannot be reconstructed afterwards.

Cross-course references

• Kubernetes: separating control-plane, data, and management networks addresses the same concerns
• Linux: out-of-band management exists precisely for the saturated-network case