Ceph's network traffic patterns

Why this matters in production

Ceph’s requirements of a network are specific and modest, and several widely repeated beliefs about them are wrong. Getting this right avoids both over-engineering and surprises at scale.

What Ceph actually uses

There is no multicast requirement, no IGMP configuration, and no broadcast domain requirement beyond ordinary IP reachability. Clusters routinely span routed layer 3 fabrics.

ss -tn state established '( sport >= :6800 and sport <= :7300 )' | wc -l
ceph config get mon ms_bind_msgr2
ceph config get mon ms_bind_msgr1

Connection scaling

Each OSD maintains connections to every other OSD it shares PGs with, which on a well-distributed cluster is most of them:

connections ≈ N_osds × (N_osds - 1)   cluster-wide, both directions counted
per OSD     ≈ N_osds - 1              plus client connections

# what one OSD actually holds
ss -tn | awk '$5 ~ /:68[0-9][0-9]|:69/ {n++} END {print n, "connections"}'

At large scale this drives file descriptor and memory requirements per OSD:

ceph config get osd ms_async_op_threads
cat /proc/$(pidof -s ceph-osd)/limits | grep -i 'open files'

Heartbeats

ceph config get osd osd_heartbeat_interval
ceph config get osd osd_heartbeat_grace
ceph config get osd osd_heartbeat_min_peers

Heartbeats are small unicast messages between OSD peers on both the public and cluster networks. They are frequent and tiny, which makes them sensitive to loss and to latency spikes but negligible in bandwidth.

What the fabric must provide

Notably absent: multicast support, a shared broadcast domain, or any layer 2 adjacency requirement.

Common misconceptions

Quiz

Production discipline

Specify full IP reachability rather than a shared VLAN in network requirements — Ceph is TCP unicast throughout and runs correctly across routed layer 3. At large scale, raise per-OSD file descriptor limits and size memory for a peer connection to nearly every other OSD.

Cross-course references

• Kubernetes: overlay networks similarly need only IP reachability, not layer 2 adjacency
• Linux: connection-count scaling drives descriptor limits in any mesh architecture