Skip to main content
RunBook Academy

CephXXVIII · Ceph NetworkingCeph Networking

The cluster network: when to have one

Advanced⏱ ~17 minceph

What you'll learn

  • Enumerate the traffic on a cluster network
  • Configure cluster_network safely
  • Weigh the benefits against the added complexity
  • Recognise cluster-network-specific failure modes

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Not yet marked complete on this device.

Why this matters in production

A cluster network is the standard recommendation for larger deployments, and it introduces a failure mode that does not exist without it. Both halves of that sentence matter, and the second is the one people are unprepared for.

What moves to the cluster network

TrafficVolume relative to client writes
Replication (primary → replicas)(size − 1)× client writes
Recovery and backfillcan vastly exceed client traffic
OSD heartbeatstiny but latency-critical
EC chunk distribution(k+m−1)/k × client writes

Client traffic, monitor traffic, and manager traffic all stay on the public network.

The volume point is the whole argument. On a size-3 pool, every client write generates two replica writes, so cluster-network traffic under steady load is roughly twice the client write volume. During recovery it is unbounded by client demand entirely.

Configuration

ceph config set global public_network  10.20.0.0/16
ceph config set global cluster_network 10.30.0.0/16

Only OSDs use the cluster network. Monitors, managers, MDS, and RGW do not — they have no OSD-to-OSD traffic.

Verify what each OSD actually bound to:

ceph osd metadata 12 | jq -r '.hb_back_addr, .hb_front_addr, .back_addr, .front_addr'

front is the public side, back is the cluster side. Confirming these after a change is worth the ten seconds.

The new failure mode

Heartbeats run on both networks. An OSD that can reach its peers on the public network but not on the cluster network is in the worst possible state: it appears alive to the monitors and cannot replicate.

Symptoms:

  • OSDs flapping — marked down, then reporting themselves up
  • heartbeat_check: no reply from ... back in OSD logs
  • Writes hanging while the cluster reports mostly healthy
  • Recovery stalling with no obvious blocker
journalctl -u ceph-osd@12 | grep -i heartbeat
ceph osd metadata 12 | jq -r '.back_addr'

The diagnosis is always the same: test connectivity on the cluster network specifically, including MTU, because the public network being fine proves nothing about it.

Is it worth it?

Yes when: recovery traffic demonstrably interferes with client latency; the cluster is large; EC pools multiply recovery volume; you have the ports and the operational maturity to run two fabrics.

No when: the cluster is small; a single fast network has ample headroom; the operations team is not equipped to troubleshoot two paths.

A single well-provisioned 25 or 100 Gb network is often better than two congested 10 Gb ones — and it is unambiguously simpler.

Quiz

Knowledge check · 4 questions

  1. Q1. Which daemons use the cluster network?

  2. Q2. An OSD reachable on the public network but not the cluster network will appear healthy to monitors while being unable to replicate.

  3. Q3. Diagnose OSD flapping after a network change.

    Following a switch firmware upgrade on the cluster network fabric, several OSDs begin flapping — marked down by peers, then reporting themselves up moments later. Client traffic is largely working but some writes hang. `ceph -s` oscillates between HEALTH_WARN and HEALTH_OK.

  4. Q4. Why might a single well-provisioned network be a better choice than two separate ones?

Passing score: 75%. Answers are checked in this browser.

Production discipline

If you run a cluster network, monitor it with synthetic checks between OSD hosts rather than waiting for flapping to reveal a problem — it carries no client traffic, so nothing else will tell you it is broken. Include both networks in every MTU and connectivity verification, and note the back_addr values somewhere durable so a comparison is available after a renumbering.

Cross-course references

  • Kubernetes: a separate storage or control-plane network has the same silent-failure property
  • Linux: a dedicated heartbeat network in a cluster manager introduces exactly this asymmetry risk