HDD — where spinning disks still belong in a Ceph cluster

Why this matters in production

HDDs are still the cheapest way to hold a petabyte, and they are still the wrong device for a VM fleet. Both statements are true, and the reason for both is the same single number: the time it takes to move a mechanical arm.

A Ceph operator needs to size HDD tiers in operations per second rather than terabytes, because capacity is the easy constraint and IOPS is the one that fails in production.

Where the time goes

A read from a spinning disk costs:

seek time      ~4-9 ms   move the head to the right track
rotational     ~2-4 ms   wait for the sector to arrive (7200 rpm ≈ 4.16 ms/rev)
transfer       ~0.1 ms   read the data
--------------------------------
random read    ~6-13 ms  →  roughly 100-170 IOPS

The transfer is negligible. Almost the entire cost is positioning, and positioning is mechanical. This is why:

• Sequential access is fast. Read a whole track, no repositioning. 150-250 MB/s from a modern enterprise drive.
• Random access is slow, and no amount of queueing fixes it. The drive can reorder requests to reduce total head travel — this is what NCQ does — but there is still one head.
• Capacity growth has not helped IOPS. A 20 TB drive has roughly the same random IOPS as a 2 TB drive from a decade ago. Per terabyte, IOPS density has fallen sharply.

Where HDD belongs in Ceph

Good fits:

• RGW data pools. Objects are large, access is sequential, and capacity dominates. This is the classic HDD workload and erasure coding makes it cheaper still.
• Backup and archive pools. Written once, read rarely, kept for years.
• CephFS bulk data pools, with the metadata pool on NVMe.

Poor fits:

• RBD for VMs or databases. Random write, latency sensitive.
• Any index or metadata pool. RGW bucket indexes and CephFS metadata are small, hot, and latency critical.
• Anything where recovery time matters. A 20 TB drive that fails takes a long time to re-replicate, and the cluster is degraded throughout.

The change that makes HDD viable

Putting the BlueStore WAL and DB on a fast device is the single largest improvement available to an HDD OSD, and it is a build-time decision:

ceph orch daemon add osd \
  host1:data_devices=/dev/sdb,db_devices=/dev/nvme0n1,osds_per_device=1

BlueStore keeps its metadata — object locations, allocation state, omap key-value data — in RocksDB. On a bare HDD every metadata read and write costs a seek that competes with data I/O. Moving RocksDB to NVMe removes that traffic from the spindle entirely.

Size the DB device deliberately. The long-standing guidance is roughly 4% of the data device for general use, and materially more for RGW workloads whose bucket indexes generate large omap. A DB device that fills spills back onto the HDD, which reintroduces the problem quietly.

Monitoring what predicts failure

smartctl -a /dev/sdX is the tool. The attributes that predict trouble on spinning media are reallocated sector count, pending sector count, and offline uncorrectable — a non-zero and rising pending count is a drive to plan the replacement of, not to watch. Rising seek error rate and command timeout are worth attention too.

Ceph surfaces some of this itself through the device health module, which stores SMART data and will predict failure if enabled.

Quiz

Production discipline

Size HDD pools in IOPS first and confirm the capacity afterwards, not the other way round. Put BlueStore WAL and DB on NVMe for any HDD OSD that will see metadata-heavy work, and size the DB device generously for RGW. Keep index and metadata pools off spinning media entirely. And factor recovery time into the drive-size decision: a 16 TB OSD failing means most of a day of degraded operation, which is a durability consideration and not only a performance one.

Cross-course references

• Linux: Part XIII (Disks and Block Devices) and Part XLI (Disk Performance).
• Ceph: Part XII (BlueStore) for WAL and DB placement in detail.
• Ceph: Part XVI (Device Classes) for keeping HDD and SSD pools separate.