Network domains — switches, uplinks, and partition behaviour

Why this matters in production

A dead host is simple: it stops, Ceph marks it down, recovery starts. A network failure is not simple, because the host is still running, still writing to its disks, and still convinced it is part of the cluster — it simply cannot be reached.

That difference is what makes network failure domains worth designing separately from power ones.

The boundaries

NIC port        one interface
NIC / bond      one card, or a bond across two cards
top-of-rack     one switch: the whole rack
uplink          the path from rack to spine
spine / core    the aggregation layer

A single top-of-rack switch is the most common network single point of failure in Ceph deployments, and it is invisible to CRUSH in exactly the way power is. Hosts that survive independently become unreachable together.

What Ceph does during a partition

flowchart TD
  A[Switch fails in rack B] --> B[OSDs in rack B unreachable]
  B --> C[Peer OSDs report failures to monitors]
  C --> D{Monitors in quorum?}
  D -->|Yes| E[Mark rack B OSDs down, then out after mon_osd_down_out_interval]
  E --> F[Recovery begins on surviving racks]
  D -->|No| G[No map updates possible — cluster stalls]

The key behaviours:

• OSDs detect peer failure through heartbeats and report to the monitors. Monitors decide, not OSDs.
• An OSD is marked down quickly, then out after mon_osd_down_out_interval (600 seconds by default), which is when recovery starts.
• The isolated OSDs keep running and keep their data. When the network returns they rejoin and peer; PGs recover rather than being rebuilt from scratch.

Public and cluster networks

Ceph supports separating client traffic (public_network) from replication and recovery traffic (cluster_network). This has a failure-domain consequence that is easy to miss: if the cluster network fails while the public network survives, OSDs cannot replicate but clients can still reach them.

The result is OSDs that appear reachable to clients and cannot complete writes, and monitors receiving conflicting failure reports from OSDs that can each reach the monitors but not each other. This is one of the more confusing failure modes in Ceph, and it argues for keeping the two networks on the same physical fabric with VLAN separation unless there is a concrete reason for physical separation.

Designing it out

• Bond NICs across two switches using MLAG, stacking, or equivalent, so a switch failure does not isolate a host.
• Spread monitors across network domains, so no single switch failure can take a majority.
• Prefer VLAN separation over physically separate fabrics for public and cluster networks, unless bandwidth genuinely requires otherwise.
• Monitor the network path itself, not only Ceph health — a link degrading to half duplex or dropping frames presents as slow ops long before it presents as a network alert.

Quiz

Production discipline

Bond NICs across two switches so a switch failure degrades bandwidth rather than isolating hosts, and spread monitors across network domains so no single switch can take a majority. Prefer VLAN separation of public and cluster traffic over physically separate fabrics unless bandwidth demands otherwise — a second fabric is a second thing that must be made redundant. Monitor the network path directly, since a degrading link surfaces as Ceph slow ops long before it surfaces as a network alert.

Cross-course references

• Ceph: Part XXX (Network Failure Behaviour) for the detailed failure modes.
• Ceph: Part CXVI (Network Partition) for the incident response.
• Ceph: Part XXIX (Network Design) for bandwidth and topology sizing.