Wrong class placement — the most common pool mistake

Why this matters in production

This is the single most common way a device-class configuration goes wrong, and it is easy to diagnose once you know the shape of it.

The symptom

ceph -s
# HEALTH_WARN Degraded data redundancy: 512 pgs undersized
ceph osd tree            # every OSD up and in
ceph pg dump_stuck undersized | head

Healthy hardware, undersized PGs, and the affected pool uses a class rule.

The diagnosis

POOL=rbd-vms
RULE=replicated_rule
ceph osd pool get ${POOL} crush_rule
ceph osd crush rule dump ${RULE}     # note the class and failure domain

# count hosts with an OSD of that class
for o in $(ceph osd crush class ls-osd nvme); do
  ceph osd find $o -f json | jq -r '.crush_location.host'
done | sort -u

Compare the host count against the pool’s size. Below it, the rule cannot be satisfied.

The two remedies

Redistribute hardware. Move devices so more hosts have the class. This delivers the protection the rule was written for and it is physical work requiring OSD recreation.

Change the rule. Lower the failure domain to osd, or drop the class, or reduce size. All three reduce protection and all three are fast.

# Substitute your own value before running:
POOL=rbd-vms

# reduce protection: class kept, host separation dropped
ceph osd crush rule create-replicated nvme_osd default osd nvme
ceph osd pool set "$POOL" crush_rule nvme_osd

The type osd variant is a legitimate stopgap for a small NVMe tier — it keeps the media guarantee while accepting that two copies may share a host. It should be a recorded decision with a plan to reverse it, not a permanent state.

The check worth automating

for cls in nvme ssd hdd; do
  n=$(for o in $(ceph osd crush class ls-osd $cls 2>/dev/null); do
        ceph osd find $o -f json | jq -r '.crush_location.host'
      done | sort -u | wc -l)
  echo "$cls: $n hosts"
done

Alert when any class used by a pool falls below that pool’s size + 1.

Quiz

Production discipline

Count class-bearing hosts as a standing metric alongside failure-domain counts, and aim for size + 1 of both — the class case is easy to miss because nobody tracks that number, and it degrades silently the first time a qualifying host is drained. Work undersized PGs in a fixed order: OSDs marked out, domain count, class host count, then crushtool --test. And treat an osd-level fallback as a recorded temporary decision, since it trades away host separation.

Cross-course references

• Ceph: Part XIV (CRUSH Failure Domains) for the general case.
• Ceph: Part XIX (PG States) for reading undersized.
• Ceph: Part XCVI (Node Maintenance) for planning drains against class rules.