Device-class rules — one cluster, several performance tiers

Intermediate⏱ ~15 minceph

What you'll learn

Create and assign class-aware CRUSH rules
Verify device class assignment across a cluster
Diagnose failures caused by insufficient hosts of a class
Plan class-tier capacity independently

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Mixed-media clusters are the norm, and device classes are how the media are kept apart. Without them, a fast pool lands on whatever CRUSH picks and its performance is set by the slowest OSD in each acting set.

Creating class rules

ceph osd crush class ls
ceph osd crush class ls-osd nvme
ceph osd crush rule create-replicated nvme_host default host nvme
ceph osd pool set cephfs_metadata crush_rule nvme_host

The class is the fourth argument. Ceph maintains a shadow tree per class automatically, so the rule descends only OSDs of that class.

ceph osd crush tree --show-shadow

Verifying class assignment

ceph-volume sets the class from the device at OSD creation, using the kernel’s rotational flag and device type. It is usually right and occasionally not:

ceph osd crush class ls-osd nvme
ceph osd crush class ls-osd ssd
ceph osd crush class ls-osd hdd
ceph osd tree | grep -c ssd

A SATA SSD behind a RAID controller presenting as rotational will be classed hdd. An NVMe device may be classed ssd depending on how it presents.

ceph osd crush rm-device-class osd.12
ceph osd crush set-device-class nvme osd.12

Changing a device class moves data

Class is part of the shadow tree membership, so changing it removes an OSD from one class tree and adds it to another. Every pool using either class recomputes placement.

On a cluster with a handful of misclassified OSDs this is minor. On a cluster where a whole tier was misclassified it is a major rebalance.

ceph config set osd osd_max_backfills 1
for o in 12 13 14; do
  ceph osd crush rm-device-class osd.$o
  ceph osd crush set-device-class nvme osd.$o
done
ceph -s

Change them in batches, waiting for active+clean between, and verify with ceph osd crush class ls-osd afterwards.

The host-count requirement

A class rule with type host needs size hosts each containing at least one OSD of that class. Total OSD count of the class is irrelevant.

# how many hosts have an nvme OSD?
for o in $(ceph osd crush class ls-osd nvme); do
  ceph osd find $o -f json | jq -r '.crush_location.host'
done | sort -u | wc -l

Twenty NVMe OSDs in two hosts cannot satisfy a size 3 NVMe host rule. This is the most common device-class failure and the count above diagnoses it in one command.

Why class tiers need independent capacity planning

Each class is effectively a separate cluster for capacity purposes. The NVMe tier fills, reaches nearfull, and blocks writes to NVMe-backed pools while the HDD tier sits at 30%.

ceph df reports per-pool available capacity correctly, since it accounts for the OSDs each pool can use. ceph status capacity summaries are cluster-wide and will look comfortable while a tier is nearly full.

So capacity monitoring must be per class:

ceph osd df tree | grep -E 'nvme|ssd|hdd'
ceph df

And expansion planning must be per class too. Adding HDD capacity does nothing for a full NVMe metadata pool, which is obvious when stated and routinely missed in budget cycles that treat storage as a single number.

The standard tiering

Class	Typical pools
`nvme`	CephFS metadata, RGW bucket index, BlueStore DB devices
`ssd`	RBD for VMs, general mixed workloads
`hdd`	RGW data, backups, archives, CephFS bulk data

Start here and deviate with a reason.

Quiz

Knowledge check · 4 questions

Q1. A cluster has 20 NVMe OSDs but a size 3 NVMe host rule leaves PGs undersized. What should be checked?

Whether the NVMe OSDs have sufficient free capacity Whether the shadow tree has been rebuilt after the last OSD addition Whether the pool pg_num is appropriate for 20 OSDs How many distinct hosts contain at least one NVMe OSD
Q2. Changing an OSD device class is a metadata-only operation with no data movement.

True False
Q3. The NVMe tier is at 88% while HDD sits at 32%. Cluster-level capacity alerts have not fired. Explain and plan.

Cluster with 12 NVMe OSDs holding CephFS metadata and the RGW bucket index pool, and 84 HDD OSDs holding bulk data. ceph status reports total cluster utilisation at 38% and health OK. ceph osd df tree shows NVMe OSDs between 85% and 90%. The next budget cycle is in four months.
Q4. Explain how device classes are maintained and why they need no manual tree work.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Verify device classes with ceph osd crush class ls-osd after every OSD addition, since controllers and enclosures can make a device present as the wrong type. Diagnose class-rule undersizing by counting hosts that contain an OSD of that class rather than counting OSDs. And monitor and plan capacity per class, because each tier is effectively a separate cluster — a cluster-wide utilisation figure will look comfortable while the NVMe tier is four months from blocking CephFS and RGW writes.

Cross-course references

Ceph: Part XVI (Device Classes) for the mechanism in full.
Ceph: Part LXIII (Capacity Management) for per-tier alerting.
Ceph: Part XIII (CRUSH Fundamentals) for shadow trees.