OSD layout: devices, hosts, and what to co-locate

Advanced⏱ ~18 minceph

What you'll learn

Apply the one-OSD-per-device model and its exceptions
Decide on DB and WAL placement
Size the DB device correctly
Balance OSD count per host against failure impact

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

The OSD layout is fixed at deployment and determines recovery behaviour, failure impact, and metadata performance for the life of the hardware. Three decisions matter: OSDs per device, DB placement, and OSDs per host.

One OSD per device

The default, and correct for HDD and most SATA SSD.

ceph orch apply osd --all-available-devices

Each device failure removes one OSD, and CRUSH’s failure model assumes device-level independence.

Several OSDs per device

Supported for high-performance NVMe, where a single OSD process cannot saturate the device:

service_type: osd
service_id: nvme-dense
placement:
  label: osd
spec:
  data_devices:
    rotational: 0
  osds_per_device: 2

Benefit	Cost
More parallelism per device	one device failure removes N OSDs
Better NVMe utilisation	more OSD processes, more memory
Higher aggregate IOPS	recovery affects more PGs at once

Justified when a single OSD leaves the NVMe substantially idle — measure before assuming. Two per device is common; more is unusual.

DB and WAL placement

BlueStore stores its RocksDB metadata on the same device as the data by default. Moving it to a faster device helps HDD OSDs substantially:

spec:
  data_devices:
    rotational: 1
  db_devices:
    rotational: 0
    limit: 2
  db_slots: 6

Configuration	Suits
DB on the data device	all-flash; simplest
DB on a fast device	HDD data; large improvement
WAL separate from DB	rarely worth it; only with three device tiers

A separate WAL is worth it only when you have a device faster than the DB device. With one flash tier, put both DB and WAL there — which happens automatically when only db_devices is specified.

Sizing the DB

The DB holds object metadata, and its size scales with object count. The historical guidance of 4% of the data device is a starting point:

16 TB HDD → 640 GB DB is generous
16 TB HDD → 60 GB DB is a common practical figure

The consequence of undersizing is spillover: metadata that does not fit on the fast device falls back to the slow one, and performance drops without an obvious cause.

ceph health detail | grep -i spillover
ceph daemon osd.12 perf dump | jq '.bluefs | {db_used_bytes, slow_used_bytes}'

slow_used_bytes above zero means spillover has occurred.

OSDs per host

OSDs per host	Failure impact	Suits
4–12	modest	most production
12–24	significant	dense storage
24+	large	archive, where rebuild time is acceptable

A host failure removes all its OSDs at once, so denser hosts mean larger recovery events and a larger capacity reserve requirement.

Quiz

Knowledge check · 4 questions

Q1. `slow_used_bytes` is above zero on several OSDs. What has happened?

The OSDs are reading from their data devices normally, since BlueStore counts every data-device read here The OSDs are approaching the full ratio and BlueStore is accounting the reserved space as slow Recovery is in progress and backfill writes are staged on the data device before the DB commits them DB spillover — RocksDB metadata did not fit on the fast device and overflowed to the slow one
Q2. Deploying multiple OSDs per device is unsupported in Ceph.

True False
Q3. Design an OSD layout for a mixed-media cluster.

New storage hosts each have 24 × 18 TB HDDs and 2 × 3.2 TB NVMe. The workload is object storage with a large number of medium-sized objects. The team plans to use all devices as OSDs.
Q4. Why do DB sizing recommendations appear as specific figures rather than a smooth percentage?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Monitor slow_used_bytes per OSD as a standing check; DB spillover degrades performance with a health warning that is easily normalised and no other symptom. Size the DB against projected object count rather than a percentage alone, and remember the layout is fixed once the OSD is created.

Cross-course references

Kubernetes: local volume topology decisions are similarly fixed at provisioning
Linux: separating a filesystem journal onto a fast device is the same optimisation