Power domains — the failure boundary CRUSH cannot see

Intermediate⏱ ~14 min

What you'll learn

Identify the power boundaries in a typical rack and row
Explain why power failure is a more common correlated cause than disk failure
Map power topology into the CRUSH hierarchy
Design pool placement so a power event is survivable

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Disks fail one at a time. Power fails in groups. A PDU trip, a breaker, a UPS that fails to transfer, a maintenance error on the wrong feed — each of these removes every server behind it simultaneously, which is precisely the correlated failure that replication is supposed to survive and often does not.

Power is also the failure domain Ceph has no visibility into. There is no command that reports which PDU a host is on.

The power hierarchy

PSU            one power supply in one server
PDU            one power strip: tens of servers
UPS            one battery system: one or more PDUs
utility feed   one incoming supply: a room or building
generator      backup for one or more feeds

Dual-PSU servers are the common mitigation at the bottom of this stack, and they only help if the two PSUs are on different PDUs fed from different UPS units. Two PSUs plugged into the same strip is a cable count, not redundancy.

Finding the boundaries

There is no Ceph command for this. The information comes from:

The facility’s power schedule or rack elevation drawings.
IPMI or BMC power-source reporting, where available.
Physically tracing cables, which is the only reliable method and is worth doing once at build time.

Record it somewhere that survives staff turnover. A per-host inventory listing rack, PDU-A feed, PDU-B feed, and switch is the artefact that makes every subsequent design question answerable.

Encoding power into CRUSH

If power boundaries do not align with racks, add an explicit level:

ceph osd crush add-bucket pdu-a1 pdu
ceph osd crush add-bucket pdu-a2 pdu
ceph osd crush move pdu-a1 rack=rack-a
ceph osd crush move pdu-a2 rack=rack-a
ceph osd crush move ceph-node-01 pdu=pdu-a1

Then write a rule that separates on pdu:

step chooseleaf firstn 0 type pdu

This is the whole technique. The pdu bucket type exists in the default type list for exactly this reason.

Why monitor placement is the first power decision

Monitors hold the cluster map and require a strict majority to make progress. With three monitors, two must be reachable; with five, three.

If monitor placement is not deliberately spread across power domains, the common outcome is that all monitors sit in the same rack because that is where the management nodes live. A single PDU trip then removes quorum, and the cluster stops serving I/O entirely — not because data is unavailable, but because no OSD can validate its view of the map.

Recovery from lost quorum is far more involved than recovery from lost OSDs: it may require extracting the monitor store from a surviving node and rebuilding, which is a longer and riskier operation than waiting for backfill. Spread monitors across power domains first, then worry about OSDs.

The design outcome

A power-aware design generally means:

Monitors on at least three distinct power domains, always.
Dual-PSU servers cross-connected to two independent PDUs.
CRUSH failure domain set to whichever of rack or pdu is the real power boundary.
A documented per-host power inventory.

None of this is expensive at build time. All of it is expensive to retrofit, because changing the CRUSH failure domain on a populated cluster moves data.

Quiz

Knowledge check · 4 questions

Q1. A PDU trips and removes 8 of 24 OSD hosts. The cluster stops serving all I/O, not just the PGs on those hosts. What is the most likely explanation?

Every PG had all three replicas on the affected hosts The cluster entered a protective read-only mode after losing a third of its OSDs Two of the three monitors were on the failed PDU, so quorum was lost min_size was set to 3, so any degraded PG stopped accepting writes
Q2. Dual-PSU servers provide power redundancy only if the two supplies are fed from independent PDUs and independent upstream sources.

True False
Q3. A cluster survives an annual power test in one rack with no impact, then loses PGs during a real UPS failure that affected two racks. Explain the difference.

30 hosts across 3 racks. CRUSH failure domain is rack, pool size 3, min_size 2. Racks A and B are fed from UPS-1; rack C is fed from UPS-2. The annual test cut power to rack A only. The real event was UPS-1 failing, which took racks A and B together. Some PGs became inactive.
Q4. Why is power the failure domain most likely to be missing from a CRUSH map, and what is the remedy?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Trace the power once at build time and record it per host, because no Ceph command will ever tell you. Place monitors across at least three power domains before considering OSD placement — losing quorum halts the cluster while losing OSDs only degrades it. Cross-connect dual-PSU servers to genuinely independent feeds. And when the power boundary does not align with racks, add a pdu level to the hierarchy rather than assuming rack separation implies power separation.

Cross-course references

Ceph: Part IX (Monitor Quorum) for why quorum loss is the worst outcome.
Ceph: Part III lesson on enterprise SSDs for what sudden power loss does to drives.
Ceph: Part CXVII (Lost Monitor Quorum) for the recovery procedure.