OSDs and devices — one, or several, per drive

Intermediate⏱ ~15 mincephcephadmceph-volume

What you'll learn

Explain the default one-OSD-per-device model
Justify multiple OSDs per device on fast media
Configure osds_per_device with cephadm
Account for the failure-domain consequence

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

The one-OSD-per-device default is right for HDD and SATA SSD and frequently wrong for high-end NVMe, where a single daemon cannot generate enough concurrent work to use the device. Getting this wrong means paying for NVMe and receiving a fraction of it.

The default

One OSD owns one block device. This is simple and it maps cleanly onto failure: the device fails, the OSD fails, CRUSH knows exactly what was lost.

ceph orch apply osd --all-available-devices
ceph-volume lvm list

For HDD and SATA SSD this is correct. A single ceph-osd process can comfortably saturate a device delivering a few hundred MB/s and a few tens of thousands of IOPS.

Several OSDs per NVMe

A modern NVMe device can sustain hundreds of thousands of IOPS across many queues. A single OSD process, with its bounded shard and thread counts, typically cannot issue that much concurrent work.

Running several OSDs on the device gives it several independent streams:

service_type: osd
service_id: nvme
placement:
  host_pattern: 'ceph-*'
spec:
  data_devices:
    rotational: false
  osds_per_device: 2

ceph orch apply -i nvme-osd-spec.yaml

Two is a common starting point; four is used on the fastest devices. Each OSD gets its own LV on the device, its own process, its own threads, and its own RocksDB.

Several OSDs on one device multiply the blast radius of that device

Four OSDs on one NVMe means one device failure removes four OSDs at once. CRUSH is unaware they share hardware — the host failure domain covers it incidentally because they are on the same host, which is usually sufficient.

What changes is the recovery arithmetic. One device failure now degrades four times as many PGs simultaneously and requires four times the backfill. Before raising osds_per_device, confirm:

the cluster has capacity to absorb losing all of them at once,
recovery from that much simultaneous loss completes in acceptable time,
and the host has CPU and memory for the additional daemons — each OSD carries its own osd_memory_target.

That last point is the one most often missed. Four OSDs per device across ten devices is forty daemons on one host.

The resource cost

Each additional OSD brings:

another process with its own memory target,
another RocksDB instance,
more PG log memory,
more CPU for peering and reporting.

So the decision is a trade: device utilisation against host resource consumption. On a host with plenty of CPU and RAM and very fast NVMe, it is clearly worthwhile. On a host that is already memory-constrained, it is not.

How to tell whether one OSD is actually the bottleneck

Do not assume; measure. The signature of a daemon-bound NVMe OSD is device utilisation well below saturation while the ceph-osd process is CPU-busy.

# on the OSD host
iostat -x 5 /dev/nvme0n1        # %util and queue depth
top -H -p $(pgrep -f 'ceph-osd -i 12')   # per-thread CPU
ceph daemon osd.12 dump_ops_in_flight | jq '.num_ops'

If %util is low, queue depth is small, and OSD threads are pinned at 100%, the daemon is the constraint and more OSDs will help. If %util is near 100% and queue depth is deep, the device is the constraint and splitting it will not.

The same measurement answers the question after the change, which is the part worth insisting on: benchmark before and after against a representative workload rather than trusting the general guidance.

Making the change

osds_per_device is set at OSD creation and changing it means destroying and recreating the OSDs on that device:

ceph orch osd rm 12 --replace
# wait for drain
ceph orch apply -i nvme-osd-spec.yaml

Do this one device at a time, waiting for active+clean between, since each recreation is a drain and a backfill.

Quiz

Knowledge check · 4 questions

Q1. When is running several OSDs on a single device the right choice?

When the device is fast enough that a single OSD process cannot saturate it Whenever the device is an SSD rather than a spinning disk, since flash rewards concurrency When the cluster needs more PGs than the current OSD count will carry within the per-OSD limit Never — CRUSH cannot express two OSDs sharing hardware, so the arrangement is unsupported
Q2. The signature of a daemon-bound NVMe OSD is low device utilisation alongside CPU-saturated OSD threads.

True False
Q3. A team plans to move from 1 to 4 OSDs per device across 10 NVMe devices per host on a 6-host cluster. Evaluate.

6 hosts, each with 10 NVMe devices, currently 1 OSD each — 60 OSDs total. Hosts have 32 cores and 128 GB RAM. Proposal moves to 4 OSDs per device: 40 OSDs per host, 240 total. osd_memory_target is the default 4 GiB. Benchmarks show current device %util around 35% with OSD threads near saturation, so the daemon-bound diagnosis is confirmed.
Q4. State what must be confirmed before raising osds_per_device, and why the change requires recreating OSDs.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Measure before splitting a device: low %util with CPU-saturated OSD threads confirms the daemon is the constraint, while a near-saturated device with a deep queue means it is not. Do the host arithmetic before choosing a number — each OSD carries its own memory target plus PG log and peering memory, and forty daemons on a 128 GiB host does not fit. And treat the change as a per-device drain and backfill, one at a time with active+clean between.

Cross-course references

Ceph: Part III (Storage Hardware) for the NVMe lesson this builds on.
Ceph: Part XI lesson on memory and threads for the resource side.
Ceph: Part XCIV (Hardware Replacement) for the drain-and-rebuild procedure.