NVMe — parallelism, not just speed

Why this matters in production

NVMe is usually described as “faster”, which undersells the change. The important difference is architectural: NVMe replaces a single 32-deep command queue behind a storage controller with many deep queues addressed directly over PCIe, one per CPU core. That changes how a Ceph OSD should be configured, not only how fast it goes.

The queue model

The per-core queue is the part that matters for a storage daemon. Each CPU core submits to its own queue and handles its own completions, so there is no shared lock and no cross-core interrupt handling on the hot path. Concurrency scales with cores rather than contending on one queue.

Why one OSD may not be enough

A ceph-osd process has a bounded amount of internal parallelism — its shards and worker threads — and on a fast NVMe device a single OSD frequently cannot issue enough concurrent work to keep the device busy. The device is idle while the daemon is the bottleneck.

The remedy is more OSDs on the same device:

ceph orch daemon add osd \
  host1:data_devices=/dev/nvme0n1,osds_per_device=2

Two or four OSDs per NVMe device is common for high-end drives. Each gets its own process, its own threads, and its own RocksDB, so the device sees several independent streams of work.

Where NVMe earns its place

In priority order, when budget is limited:

1. BlueStore WAL and DB for HDD OSDs. The highest return on a small number of devices. Removes all metadata seeks from spindles.
2. Metadata pools. CephFS metadata and RGW bucket indexes are small, hot, and latency critical. A handful of NVMe devices transforms them.
3. Latency-sensitive RBD pools. Databases and anything with a tight commit budget.
4. General OSD data. Excellent, and the most expensive way to spend the budget per terabyte.

That ordering is worth following literally. Ten NVMe devices placed at points one and two often do more for a cluster than sixty placed at point four.

Monitoring NVMe health

nvme smart-log /dev/nvme0n1
nvme error-log /dev/nvme0n1

The fields to watch are percentage_used (the drive’s own estimate of consumed endurance, where 100 does not mean failure but does mean plan replacement), media_errors, critical_warning, and temperature. Thermal throttling is a real effect on dense NVMe deployments and presents as unexplained periodic latency spikes — worth checking before blaming Ceph.

Quiz

Production discipline

Spend NVMe budget in order: BlueStore WAL and DB for HDD OSDs first, then metadata pools, then latency-sensitive data pools, and only then general capacity. Expect a single OSD process not to saturate a high-end device, and raise osds_per_device deliberately after confirming the cluster can absorb losing all the OSDs on one device at once. And plan CPU and network alongside NVMe, because at microsecond device latency they become the components that decide what the client actually sees.

Cross-course references

• Ceph: Part XII (BlueStore) for WAL and DB device roles.
• Ceph: Part XXIX (Network Design) for why NVMe clusters need faster links.
• Ceph: Part XVI (Device Classes) for separating NVMe from HDD in CRUSH.