CRUSH rules that select by class

Intermediate⏱ ~15 mincephcrushtool

What you'll learn

Write and create class-aware CRUSH rules
Explain the shadow tree mechanism
Compute the host requirement for a class rule
Verify a pool is genuinely placed on the intended class

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Pinning a pool to a media class is the single most effective performance decision available, because it removes the possibility of a slow device appearing in a fast pool’s acting set.

The rule

ceph osd crush rule create-replicated nvme_host default host nvme
ceph osd crush rule dump nvme_host

rule nvme_host {
  id 3
  type replicated
  step take default class nvme
  step chooseleaf firstn 0 type host
  step emit
}

take default class nvme is the only difference from a plain rule.

The shadow tree

Ceph maintains a parallel hierarchy per class:

ceph osd crush tree --show-shadow

ID   CLASS  WEIGHT    TYPE NAME
-2   nvme    14.552   root default~nvme
-8   nvme     7.276       host ceph-01~nvme
 3   nvme     3.638           osd.3
 5   nvme     3.638           osd.5
-1          291.040   root default
-7           72.760       host ceph-01
 3   nvme     3.638           osd.3
 4   hdd      7.276           osd.4

default~nvme contains only NVMe OSDs, with host buckets whose weights reflect only those OSDs. take default class nvme is take default~nvme.

The shadow trees are maintained automatically as OSDs are created, removed, or reclassified.

The requirement is hosts with that class, not OSDs of that class

A size 3 rule with type host and class nvme needs three distinct hosts, each containing at least one NVMe OSD.

Thirty NVMe OSDs across two hosts do not satisfy it. Three NVMe OSDs across three hosts do.

for o in $(ceph osd crush class ls-osd nvme); do
  ceph osd find $o -f json | jq -r '.crush_location.host'
done | sort -u | wc -l

That count against the pool’s size is the check, and it is the most common reason a class rule leaves PGs undersized. Plan flash purchases to spread across hosts rather than to concentrate, even when concentrating is cheaper per device.

Verifying placement

Creating the rule and assigning it is not proof:

PGID=4.1a                      # one PG from the listing below
ACTING="7 19 31"               # its acting set, from ceph pg map

ceph osd pool set cephfs_metadata crush_rule nvme_host
ceph pg ls-by-pool cephfs_metadata | head -3
ceph pg map "${PGID}"
for o in ${ACTING}; do
  ceph osd crush get-device-class "osd.$o"
done

Every OSD in the acting set should report nvme. Sampling a few PGs confirms the rule is doing what its name claims.

Why shadow tree weights differ and what that means for capacity

The default root’s weight is the sum of every OSD. The default~nvme root’s weight is the sum of only the NVMe OSDs.

So each class root describes an independent capacity pool. A pool on nvme_host can use only the NVMe capacity, and reaches nearfull when those OSDs do — regardless of vast free space on HDD.

ceph df accounts for this correctly, reporting per-pool available capacity from the OSDs the pool can actually use. Cluster-level summaries do not.

The planning consequence is that each class needs its own capacity forecast and its own alerting. A budget conversation about “adding storage” that does not specify which tier is not a plan, because adding HDD does nothing for a full metadata pool and adding NVMe does nothing for a full archive.

The standard set

ceph osd crush rule create-replicated nvme_host default host nvme
ceph osd crush rule create-replicated ssd_host  default host ssd
ceph osd crush rule create-replicated hdd_host  default host hdd

Three rules cover most clusters. Add rack-level variants where the topology supports them.

Quiz

Knowledge check · 4 questions

Q1. A cluster has 30 NVMe OSDs but a size 3 NVMe host rule cannot be satisfied. What is the requirement it fails?

Thirty distinct hosts, one for each NVMe OSD in the cluster At least three NVMe OSDs inside every host under the root An nvme device class declared explicitly in the CRUSH map Three distinct hosts each containing at least one NVMe OSD
Q2. A pool on an NVMe class rule can use free capacity on HDD OSDs when the NVMe tier fills.

True False
Q3. Verify that a pool moved to an nvme class rule is actually placed on NVMe, and design a standing check.

CephFS metadata pool was moved from hdd_host to nvme_host last week. Latency improved but the team wants proof rather than inference, and a standing check so a future misclassification or rule reassignment is caught. Cluster has 96 OSDs of which 12 are NVMe across 6 hosts.
Q4. Explain what take default class nvme resolves to and how the shadow tree is maintained.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Count hosts containing an OSD of the class rather than counting OSDs of the class — that is the requirement a class rule actually imposes, and spreading flash across hosts matters more than concentrating it cheaply. Verify placement by resolving real acting sets to their device classes rather than by reading the rule, and make that a standing check. And forecast and alert on capacity per class, since each shadow root is an independent capacity pool that cluster-level summaries hide.

Cross-course references

Ceph: Part XV (CRUSH Maps and Rules) for rule creation.
Ceph: Part XVI lesson on wrong class placement for diagnosis.
Ceph: Part LXIII (Capacity Management) for per-tier forecasting.