When CRUSH cannot satisfy the rule

Intermediate⏱ ~16 mincephcrushtool

What you'll learn

Recognise the signature of an unsatisfiable CRUSH rule
Diagnose which constraint cannot be met
Choose between changing the rule and changing the topology
Prevent the situation with pre-application validation

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

undersized with no failed hardware is one of the clearest diagnostic signals in Ceph, and it is frequently misread as a bug because CRUSH gives no explanation.

The signature

ceph -s
# HEALTH_WARN Degraded data redundancy: N pgs undersized
ceph pg dump_stuck undersized
ceph osd tree            # all OSDs up and in

All OSDs healthy, PGs undersized. CRUSH tried, exhausted its retries, and returned a short acting set without saying why.

Diagnosing the constraint

Read the rule, then count what it needs:

ceph osd pool get rbd-vms crush_rule
ceph osd crush rule dump rack_rule
ceph osd tree | grep -c '^-.*rack'

Rule requirement	Count needed	Check
`type host`, `size 3`	3 hosts with usable OSDs	`ceph osd tree`
`type rack`, `size 3`	3 racks with usable OSDs	count rack buckets
`class nvme`, `type host`	3 hosts each with an NVMe OSD	`ceph osd crush class ls-osd nvme`

The device-class case is the one that catches people: a cluster with plenty of NVMe concentrated in two hosts cannot satisfy a size 3 NVMe host rule, however many NVMe OSDs exist.

Marked-out OSDs do not count toward the rule

An OSD that is up but out is excluded from CRUSH selection. So a rule that was satisfiable becomes unsatisfiable the moment enough OSDs are drained.

The common case is maintenance: draining a whole rack under a rack-level rule with exactly three racks leaves two, and every PG in the pool goes undersized.

ceph osd tree | grep -c 'up.*1.00000'
ceph osd df tree | awk '$NF == "0" {print}'

This is not a fault — the rule is being enforced correctly — but it means domain-level maintenance and domain-level rules have to be planned together, and it is the strongest argument for having size + 1 domains.

The options

Change the rule. Lower the failure domain to a level the topology satisfies. Fast, and reduces protection.

ceph osd crush rule create-replicated host_rule default host
ceph osd pool set rbd-vms crush_rule host_rule

Change the topology. Add a domain — a rack, a host, or NVMe OSDs in a third host. Slower, and delivers the protection that was intended.

Change size. Reducing size to match the available domains preserves the failure domain at lower redundancy. Rarely the right answer for production data.

The choice depends on whether the protection was a requirement or an aspiration, which is a question for the design document rather than for the cluster.

Why choose_total_tries occasionally matters and usually does not

CRUSH retries when a selection lands on an unusable item, bounded by choose_total_tries at 50.

In a genuinely marginal topology — a rule that is satisfiable but only just, such as a size 3 host rule on exactly three hosts of very different sizes — the pseudo-random selection can fail to find a valid combination within 50 tries for a small number of PGs.

ceph osd crush tunables optimal
crushtool -i /tmp/cm.bin --test --rule 0 --num-rep 3 \
    --show-bad-mappings --set-choose-total-tries 100

Raising it can resolve those few PGs. But if a rule needs it raised substantially, that is evidence the topology is too marginal for the rule rather than a tuning opportunity — and the number of retries per placement is a cost paid on every computation.

The honest reading of needing more tries is usually “add a domain”.

Preventing it

Validate before applying, every time:

crushtool -i /tmp/cm.new --test --rule 1 --num-rep 3 --show-bad-mappings

Zero bad mappings means the rule is satisfiable against that map. It is a five-second check that converts this entire class of incident into a desk finding.

Quiz

Knowledge check · 4 questions

Q1. A cluster reports undersized PGs with every OSD up and in. What is the most likely explanation?

A monitor is out of quorum, so the OSD map cannot be updated with the current acting sets The CRUSH rule requires more failure domains than the topology provides The pool's pg_num is too high for the OSD count, so some PGs get no complete acting set The balancer has written upmap entries that place PGs outside the pool's failure domain
Q2. An OSD that is up but marked out still counts toward satisfying a CRUSH rule.

True False
Q3. A pool is moved to an NVMe device-class rule and immediately reports a third of its PGs undersized. All NVMe OSDs are healthy.

Cluster of 24 hosts. 30 NVMe OSDs exist, but they are concentrated: 15 in host ceph-01, 10 in ceph-02, and 5 in ceph-03. All other hosts have only HDD. The new rule is take default class nvme, chooseleaf firstn 0 type host, for a size 3 pool. Roughly a third of PGs are undersized.
Q4. Give the diagnostic sequence for undersized PGs on a cluster with no failed hardware.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Read undersized on healthy hardware as an unsatisfiable rule and diagnose it by counting buckets at the rule’s level — remembering that OSDs marked out do not count, and that a class-aware rule needs hosts with that class rather than merely enough OSDs of it. Run crushtool --test --show-bad-mappings before applying any rule, which turns this whole class of incident into a five-second desk check. And treat a need to raise choose_total_tries as evidence the topology is too marginal, not as a tuning success.

Cross-course references

Ceph: Part XVI (Device Classes) for the class-rule case.
Ceph: Part XIX (PG States) for reading undersized correctly.
Ceph: Part XCVI (Node Maintenance) for planning drains against rules.