Choosing a device class for a workload

Intermediate⏱ ~17 minceph

What you'll learn

Match device classes to workload characteristics
Compute the device count a workload requires
Recognise a mismatched pairing
Plan a mixed-class cluster

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Device class is the single hardware decision with the largest effect on what a pool can do, and it cannot be changed without moving all the data.

Matching class to workload

Workload	Characteristic	Class
VM root disks	small random I/O, latency-sensitive	SSD or NVMe
Database volumes	small random writes, fsync-heavy	NVMe with PLP
CephFS metadata	tiny operations, very latency-sensitive	NVMe
RGW bucket index	small random, high rate	NVMe
RGW data, large objects	sequential, throughput-oriented	HDD
Backup targets	sequential write, rare read	HDD
Video or archive	sequential, cold	HDD

The two entries most often got wrong are CephFS metadata and the RGW bucket index: both are small pools with disproportionate latency impact, and both are frequently left on the same HDDs as the bulk data.

RULE=replicated_rule
ceph osd pool get cephfs_metadata crush_rule
ceph osd crush rule dump ${RULE} | grep -A2 class

Computing the device count

A workload’s IOPS requirement divided by per-device IOPS gives the device count, before replication:

required: 40,000 random write IOPS
size=3 → 120,000 device-level write operations
HDD at 150 IOPS → 800 devices
NVMe at 300,000 IOPS → 1 device (but you need more for capacity and durability)

That arithmetic is the argument against HDD for random workloads: the device count needed for IOPS vastly exceeds the count needed for capacity.

# what the current pool can do
ceph osd pool stats rbd-vms
ceph osd df | grep -c hdd

Recognising a mismatch

Symptom	Likely mismatch
Latency at the HDD floor with low utilisation	random workload on HDD
Devices at 100% util with modest IOPS	HDD serving random I/O
Excellent throughput, poor latency	sequential-capable hardware, random workload
CephFS operations slow, data fast	metadata pool on the wrong class

iostat -x 1 5
# %util near 100 with r/s + w/s in the low hundreds = HDD at its random limit

Planning a mixed cluster

ceph osd crush class ls
ceph osd crush rule create-replicated nvme-rule default host nvme
ceph osd crush rule create-replicated hdd-rule default host hdd

ceph osd pool set cephfs_metadata crush_rule nvme-rule
ceph osd pool set cephfs_data crush_rule hdd-rule

The classes are assigned automatically from device type and can be overridden:

ceph osd crush set-device-class nvme osd.12
ceph osd crush rm-device-class osd.12

Changing a device class moves data for every pool whose rule selects it, so it is a planned operation rather than a correction.

Quiz

Knowledge check · 4 questions

Q1. Why does an HDD cluster sized for a random IOPS requirement end up with far more capacity than needed?

HDDs must be kept well below their rated capacity to hold random IOPS steady, so each drive contributes only part of its size Replication multiplies the capacity requirement threefold at size=3, so a cluster sized for its IOPS requirement is oversized on capacity by that factor Per-drive IOPS is set by seek and rotation and has not scaled with capacity, so the drive count for IOPS far exceeds the count for capacity Ceph reserves the space between the nearfull and full ratios on every OSD, so usable capacity per drive is far below its rated size
Q2. CephFS metadata pools are small enough that their device class rarely matters.

True False
Q3. Diagnose a slow CephFS deployment.

A CephFS deployment shows good sequential throughput but very slow directory listings and file creation. All pools use the same HDD-backed CRUSH rule.
Q4. Which two pools are most commonly left on the wrong device class, and why does it matter?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Put the CephFS metadata pool and the RGW bucket index on NVMe; they are small enough that the cost is a few drives and every operation in those services waits on them. Size random-IOPS workloads by dividing the requirement by per-device IOPS — the arithmetic is what rules out HDD before the discussion becomes about price.

Cross-course references

Kubernetes: matching storage class to workload characteristics is the same decision
Linux: separating a database’s WAL onto faster media follows identical reasoning