The lifecycle of a device class assignment

Intermediate⏱ ~15 mincephceph-volume

What you'll learn

Trace the class assignment through an OSD lifecycle
Identify the points where drift occurs
Preserve class assignments through hardware replacement
Build class verification into operational runbooks

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Class assignment is set once and then forgotten, which is exactly the shape of thing that drifts. The drift is silent and it surfaces months later as an unsatisfiable rule or an unexplained latency.

The lifecycle

creation     ceph-volume reads the device, sets a class
running      class stored in the CRUSH map, persists across restarts
replacement  new device → new OSD → class read again from the new device
removal      OSD leaves both the default tree and its class shadow tree

The replacement step is where drift enters. A replacement device of a different type, or behind a different controller, gets classed from what it reports — which may not match what the OSD it replaced was classed as.

Verify class after every replacement, in the runbook

# after replacing osd.12
ceph osd crush get-device-class osd.12
ceph osd crush class ls-osd nvme | grep -w 12

The failure this catches: a host of NVMe OSDs where one replacement was a SATA SSD, or where a replacement went through a controller that reports rotational. The OSD works, the pool works, and one acting set member in every PG that OSD serves is now the slow one.

Add the check as a line in the replacement runbook rather than relying on an audit to find it later. Two commands, once per replacement.

Class through a cephadm replacement

ceph orch osd rm 12 --replace
# wait for drain
# physically replace the device
ceph orch apply -i osd-spec.yaml
ceph osd crush get-device-class osd.12

--replace keeps the OSD id reserved so the new device takes the same id, which keeps CRUSH churn minimal. The class is still read fresh from the new device.

Where else drift appears

A spec without device filters. An OSD created by --all-available-devices gets whatever class its device reports, which may not be the tier intended for that slot.
Manual set-device-class never repeated. If someone corrected a class by hand, a later replacement of that device reverts to the automatic value.
Custom classes. Never assigned automatically, so every new OSD needs the manual step.

Why class changes are recorded in the CRUSH map, not on the device

The class lives in the CRUSH map within the osdmap, not in any metadata on the device itself.

That has two consequences. A manual set-device-class survives OSD restarts and host reboots, because it is cluster state — which is what makes hand corrections stick.

And it does not survive OSD recreation, because a new OSD is a new entry with a class read fresh from the device. So a cluster where three OSDs were hand-corrected years ago will silently revert those three the next time their devices are replaced, with nothing recording that the correction was ever needed.

The durable answer is either to fix the underlying reporting — HBA mode rather than RAID, for instance — or to script the correction as part of the OSD creation runbook so it is applied every time rather than remembered.

Building it into runbooks

Add to the OSD creation and replacement procedures:

expected=nvme
actual=$(ceph osd crush get-device-class osd.$id)
[ "$actual" = "$expected" ] || {
  echo "class mismatch: expected $expected got $actual"
  ceph osd crush rm-device-class osd.$id
  ceph osd crush set-device-class $expected osd.$id
}

An explicit expected value per host or per slot, checked at creation, removes this entire class of drift.

Quiz

Knowledge check · 4 questions

Q1. An OSD class was corrected by hand three years ago. Its device is now replaced. What happens to the class?

The manual correction persists, since the class is cluster state rather than device metadata The new OSD gets a class read fresh from the device, reverting the correction The orchestrator reapplies the previous class when it recreates the OSD under the reserved id The OSD refuses to start until an operator assigns a class explicitly to the new device
Q2. Verifying device class should be a step in the OSD replacement runbook rather than a periodic audit.

True False
Q3. Design the class verification step for a cluster with three tiers and frequent drive replacements.

Cluster with 12 NVMe, 36 SSD, and 96 HDD OSDs across 12 hosts. Drive replacements happen roughly monthly. Hosts are homogeneous within a tier: 4 hosts are all-NVMe, 4 are all-SSD, 4 are all-HDD. Two past incidents involved a replacement being classed wrongly and serving the wrong tier for weeks.
Q4. Name three places device class drift enters a cluster.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Put class verification in the OSD creation and replacement runbooks rather than relying on periodic audits, since two commands at the moment of risk beat discovering a wrong-tier OSD months later. Make the expected class explicit — host labels work well on homogeneous hosts — and have the check correct rather than merely report. And prefer fixing the underlying reporting, such as putting a controller into HBA mode, over repeating a manual correction that will revert at the next replacement.

Cross-course references

Ceph: Part XCIV (Hardware Replacement) for the replacement procedure.
Ceph: Part XVI lesson on classification for how classes are assigned.
Ceph: Part XLIX (cephadm) for OSD specs and host labels.