Skip to main content
RunBook Academy

← All checklists in Ceph

Quarterlyceph-crush-review

CRUSH / Failure Domain Review Checklist

6 items ·3 critical ·3 warn ·0 info

Purpose

The CRUSH map is the cluster’s model of the building. Hardware moves, racks get consolidated, a host is re-added under the default root after a rebuild - and the map quietly stops matching the room. Nothing alerts on this. The cluster stays HEALTH_OK while placing what it believes are three independent replicas on three OSDs in one rack. This quarterly review is how the drift gets caught.

Order of work

The list runs in three passes. The shape pass is physical-vs-crush, replica-host-step and orphaned-buckets: read the tree against the rack diagram, confirm the rule actually separates replicas at the level you think it does, and clear the buckets left behind by decommissions. The distribution pass is weight-fairness and balancing-mode, which decide whether the capacity inside that shape is used evenly. The binding pass is rule-consistency, which checks the rules back against the pools that reference them - a correct rule nothing points at protects nothing.

Before you start

Have the current rack and power-feed diagram, the pool-to-rule mapping, and the device classes in use. The review is a comparison against the physical layout, so without the diagram there is nothing to compare to.

Verification

The review passes when every critical item is checked and any discrepancy between the map and the room has a ticket. Mark an item N/A rather than leaving it unchecked when it does not apply to this cluster.

Escalation

A rule that does not separate replicas at the intended failure domain is a live durability defect. Escalate it the day you find it.

Critical3 items

  1. ceph osd tree
  2. ceph osd crush rule dump
  3. ceph osd pool ls detail

Warning3 items

  1. ceph osd crush dump
  2. ceph osd crush dump
  3. ceph balancer mode