CephXXVIII · Ceph NetworkingCeph Networking
The public network: what crosses it
What you'll learn
- Enumerate the traffic types on the public network
- Configure public_network correctly
- Predict the impact of public network problems
- Size the public network against client demand
Prerequisites
None — start here.
Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18
Why this matters in production
The public network is the one clients see. If it is undersized or
unreliable, every symptom you get looks like “Ceph is slow” regardless of
how healthy the storage layer is, and the diagnosis is easy to miss
because ceph -s will happily report HEALTH_OK throughout.
What rides it
| Traffic | Endpoints |
|---|---|
| Client reads and writes | librados/librbd/kernel client ↔ OSDs |
| Monitor conversations | clients ↔ MONs, daemons ↔ MONs |
| OSD map distribution | MONs → OSDs and clients |
| Manager traffic | daemons ↔ MGR, dashboard, metrics |
| RGW ↔ OSDs | the gateway is a client |
| MDS ↔ clients and OSDs | CephFS metadata operations |
Note the fourth and fifth rows. RGW and MDS are RADOS clients, so their traffic to OSDs is public-network traffic even though they are cluster components.
Configuration
ceph config set global public_network 10.20.0.0/16
ceph config get mon public_networkThe value is a CIDR, not an address. Each daemon binds to whichever of its local addresses falls inside that range. Multiple ranges are comma separated:
ceph config set global public_network 10.20.0.0/16,10.21.0.0/16Monitors are the special case: their addresses are recorded in the monmap, and changing a monitor’s address is a monmap edit rather than a config change.
ceph mon dumpIf there is no cluster network
Without cluster_network set, everything rides the public network —
client I/O, replication, recovery, and heartbeats together. That is a
perfectly valid configuration for small clusters and it is the default.
The consequence is that recovery traffic directly contends with client
traffic on the same links.
Failure behaviour
| Problem | Symptom |
|---|---|
| Congestion | client latency rises; cluster health stays OK |
| Packet loss | client retries, occasional slow ops |
| MTU mismatch | large operations hang while small ones work |
| Partition between clients and OSDs | clients hang; cluster looks healthy |
| Partition between OSDs and MONs | OSDs marked down despite being alive |
The last two are the ones that waste hours: the cluster reports itself healthy while clients are down, or reports OSDs down that are running perfectly. Both are network diagnoses, not storage ones.
Quiz
Knowledge check · 4 questions
Q1. An RGW gateway communicates with OSDs. Which network does that traffic use?
Q2. Setting public_network to a CIDR causes each daemon to bind to whichever of its local addresses falls within that range.
Q3. Diagnose a latency complaint that the cluster does not see.
One application team reports RBD latency five times higher than normal. `ceph -s` is HEALTH_OK, `ceph osd perf` shows no outliers, and other teams on the same cluster report normal performance.
Q4. Why does an OSD fail to start with a bind error after its host is moved to a new VLAN?
Passing score: 75%. Answers are checked in this browser.
Production discipline
Measure and record baseline client-to-OSD round-trip times so a latency complaint can be compared against a known-good figure rather than against intuition. Include the public network CIDR in host provisioning checks; a renumbered host that no longer matches is a startup failure that looks alarming and has a trivial cause.
Cross-course references
- Kubernetes: pod-to-service network path issues present identically as application slowness
- Linux: binding a service to an address that no longer exists is the same class of failure