Mixed-storage clusters — safe when separated, dangerous when not

Intermediate⏱ ~16 minceph

What you'll learn

Explain why unseparated mixed media performs at the slowest device
Design a tiered cluster with appropriate rules
Detect a pool that spans device classes
Migrate an unseparated cluster to class-based tiers

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Mixing media is the normal way to build a cost-effective cluster. Mixing media in a single pool is a way to pay for NVMe and receive HDD latency, and the mechanism is worth understanding precisely.

Why unseparated mixing fails

A replicated write is acknowledged only when every OSD in the acting set has committed. So for a PG whose acting set contains one HDD and two NVMe OSDs:

NVMe commit:  ~0.1 ms
NVMe commit:  ~0.1 ms
HDD commit:  ~10   ms
client sees: ~10   ms

The fast devices contribute nothing to write latency. Reads served from the primary are fast only if the primary happens to be NVMe, which is a coin toss.

Worse, capacity distribution follows weight, so a small number of fast devices in a large HDD pool hold a small share of the data — meaning most reads land on HDD anyway.

Detecting a pool that spans classes

# POOL is the pool to sample, from `ceph osd pool ls`; substitute your own:
POOL=rbd-vms

for pg in $(ceph pg ls-by-pool "$POOL" -f json | jq -r '.pg_stats[].pgid' | head -30); do
  for o in $(ceph pg map $pg -f json | jq -r '.up[]'); do
    ceph osd crush get-device-class osd.$o
  done | sort -u | tr '\n' ' '
  echo "  <- $pg"
done

Any PG reporting more than one class is a PG whose latency is set by the slowest member. If the output shows mixed classes across the sample, the pool is unseparated.

This can happen by design on legacy clusters and by accident when a pool’s rule was never updated after fast devices were added — which is the more common case and the more frustrating one, because the hardware was bought specifically to make that pool faster.

Designing the tiers

ceph osd crush rule create-replicated nvme_host default host nvme
ceph osd crush rule create-replicated ssd_host  default host ssd
ceph osd crush rule create-replicated hdd_host  default host hdd

ceph osd pool set cephfs_metadata crush_rule nvme_host
ceph osd pool set rgw_index       crush_rule nvme_host
ceph osd pool set rbd_vms         crush_rule ssd_host
ceph osd pool set rgw_data        crush_rule hdd_host

Each pool now sits on uniform media, so its latency is that media’s latency.

Migrating an unseparated cluster

The migration is a rule reassignment per pool, and it moves data:

ceph config set osd osd_max_backfills 1
ceph config set osd osd_recovery_sleep 0.05

# smallest and most latency-critical pool first
ceph osd pool set cephfs_metadata crush_rule nvme_host
# wait for active+clean
ceph osd pool set rbd_vms crush_rule ssd_host
# wait
ceph osd pool set rgw_data crush_rule hdd_host

Order matters: doing the metadata pool first delivers the largest improvement for the least movement.

Why a small fast tier in a big slow pool is worse than useless

Consider adding 6 NVMe OSDs to a 90-OSD HDD pool without changing the rule.

CRUSH distributes by weight, so the NVMe devices receive roughly their weight share — perhaps 4% of the data. The other 96% stays on HDD.

Write latency is unchanged, because every acting set still contains HDDs. Read latency improves for the 4% of objects whose primary is NVMe, which is invisible in aggregate.

Meanwhile the NVMe devices are being consumed by a workload that cannot benefit from them, and the same six devices used as BlueStore DB devices for the HDD OSDs, or as a dedicated metadata pool, would transform the cluster.

This is the concrete argument for planning flash placement before purchase: the same money produces a large improvement or none at all depending entirely on where the devices are pointed.

The checks

Does every pool’s rule name a class?
Do sampled acting sets contain a single class?
Is each class’s capacity tracked separately?
Are flash devices placed where the metadata is, not where the bytes are?

Quiz

Knowledge check · 4 questions

Q1. Six NVMe OSDs are added to a 90-OSD HDD pool without changing the CRUSH rule. What is the effect on write latency?

Essentially none, because every acting set still contains HDDs and the write waits for all of them It improves in proportion to the NVMe capacity added, since CRUSH weights the fast devices higher It improves for every write, because the primary is now more likely to be one of the fast devices It worsens, because a mixed-speed acting set makes the slower members miss the commit and retry
Q2. A pool whose sampled PG acting sets contain more than one device class has latency set by its slowest member.

True False
Q3. A cluster added 12 NVMe OSDs a year ago to improve CephFS metadata performance, with no measurable result. Diagnose.

96 HDD OSDs and 12 NVMe OSDs across 8 hosts. CephFS metadata pool, CephFS data pool, and an RGW data pool all use the default replicated_rule with host separation and no device class. The NVMe was purchased specifically to accelerate metadata operations. Metadata latency is unchanged from before the purchase.
Q4. Explain why capacity distribution makes a small fast tier in a large slow pool ineffective even for reads.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Give every pool a rule that names a device class, and verify by sampling acting sets rather than by reading rules — a pool spanning classes commits at the slowest member’s speed and no amount of fast hardware changes that. When adding flash to an existing cluster, decide where it points before buying: metadata pools and BlueStore DB devices transform a cluster, while the same devices dropped into a large slow pool deliver nothing measurable.

Cross-course references

Ceph: Part XII (BlueStore) for the DB device alternative.
Ceph: Part XVI lesson on performance impact for the measurements.
Ceph: Part III (Storage Hardware) for choosing the media.