Switch failure: what Ceph sees

Why this matters in production

Switch failures happen, and the difference between a non-event and a major incident is entirely in how the hosts were connected. Knowing which outcome your design produces is something to establish before the switch fails, not during.

The three outcomes

Dual-homed with MLAG — invisible. Each host has bond members on two switches. One switch fails, each bond drops to one member, throughput halves, and Ceph sees nothing at all. No OSDs down, no recovery, no incident.

Single ToR, rack is a CRUSH failure domain — disruptive but safe. All hosts in the rack go offline together. With failure_domain=rack and enough racks, every PG keeps at least min_size replicas and the cluster keeps serving degraded.

ceph -s
  osd: 120 osds: 100 up, 120 in
  pgs: 4096 active+undersized+degraded

Client I/O continues. Recovery starts after the down-out interval unless you prevent it.

Single ToR, failure domain smaller than the switch — catastrophic. With failure_domain=host and all three replicas of many PGs in one rack, a switch failure takes multiple replicas of the same PGs simultaneously. PGs go inactive. Clients hang.

Responding while it is happening

# stop the cluster from redistributing data for a switch that will return
ceph osd set noout

# assess what is actually affected
ceph -s
ceph osd tree | grep down
ceph pg dump_stuck inactive

noout first, always. A switch outage of thirty minutes does not warrant rebalancing 20 hosts’ worth of data, and the rebalance will have to be undone when the switch returns.

Then determine whether any PGs are inactive. If they are, the failure domain does not match the physical topology, and that is the finding to record for afterwards.

When the switch returns

ceph osd tree                # confirm OSDs are back up
ceph -s                      # watch degraded counts fall
ceph osd unset noout         # only after stability is confirmed

Expect recovery proportional to how long the outage lasted — writes that happened while the rack was offline must be replicated to it. A short outage recovers from the PG log; a long one triggers backfill.

Quiz

Production discipline

Establish and document what a single switch failure does to your cluster before it happens, and make noout the first action in the switch-failure runbook. Where the answer is “PGs go inactive,” treat that as a design finding requiring a failure-domain change rather than as an operational procedure to be practised.

Cross-course references

• Kubernetes: node failures crossing a topology boundary have the same design-dependent severity
• Linux: multipath configuration determines whether a fabric failure is visible to the filesystem