Routed network designs for Ceph

Why this matters in production

Large layer 2 domains do not scale gracefully — broadcast traffic, spanning tree convergence, and MLAG complexity all grow with them. Routed designs, with layer 3 terminating at the leaf or the host, are how large clusters are built now, and Ceph handles them well provided a few specifics are right.

Ceph does not require layer 2

Ceph daemons communicate over TCP with no reliance on broadcast, multicast, or link-local discovery. Hosts in different subnets work without special configuration, as long as they can route to each other.

# hosts in several subnets, all covered
ceph config set global public_network 10.20.1.0/24,10.20.2.0/24,10.20.3.0/24
ceph config set global cluster_network 10.30.1.0/24,10.30.2.0/24,10.30.3.0/24

Each daemon matches its local address against the list. One rack per subnet is a common and clean arrangement.

What routing buys

• Small broadcast domains — a rack, not a cluster
• No spanning tree across the fabric — ECMP instead, with fast reconvergence
• Predictable failure behaviour — a link failure is a routing event, not a topology recalculation
• Horizontal scale — adding a rack adds a subnet, not more broadcast domain

What it requires

Routing convergence must be fast. Ceph marks OSDs down after about 20 seconds of missed heartbeats, so routing reconvergence needs to be well inside that. BGP with BFD achieves sub-second; slower IGP tuning may not.

Latency must stay low. Each routed hop adds latency to every operation. In a leaf-spine fabric that is one or two hops and is fine; a design where OSD traffic traverses several routers is not.

Monitor addresses must be stable and reachable. Monitors are identified by address in the monmap, and every client and daemon must reach every monitor.

Monitors in a routed design

ceph mon dump

Monitor addresses are recorded in the monmap, not derived from public_network at runtime. Placing monitors in different subnets is fine and is good practice for fault isolation, but:

• Every client must route to every monitor address
• Changing a monitor’s address requires a monmap update, not a config change
• Firewall rules must permit monitor ports from every client network

Some designs use anycast or a loopback address advertised by BGP for monitors, which keeps the monmap stable across physical moves.

Quiz

Production discipline

Measure routing reconvergence at commissioning and compare it explicitly against osd_heartbeat_grace; that single number determines whether ordinary link failures stay invisible to Ceph. Document the monitor addressing scheme and the procedure for changing it, since it is the part of a routed design that is most awkward to alter later.

Cross-course references

• Kubernetes: routed pod networking without overlays has the same scaling rationale
• Linux: BGP to the host with FRR is the same pattern used for anycast services