Contention between workloads on shared OSDs

Advanced⏱ ~18 minceph

What you'll learn

Explain why per-pool tuning provides little isolation
Identify the mechanisms that do isolate
Diagnose cross-tenant interference
Design for isolation where it is required

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Two pools on the same OSDs compete for the same devices. Most per-pool settings do not change that, and knowing which mechanisms actually isolate prevents a lot of ineffective tuning.

Why per-pool tuning does not isolate

ceph osd pool get rbd-vms recovery_priority
ceph osd pool get rbd-vms pg_num

Pool settings affect placement, redundancy, and recovery ordering. None of them reserve device capacity — the OSD serves operations from all its PGs through the same queues and the same device.

pool A: 40k IOPS of random write
pool B: expects 5 ms p99
  → both hit the same OSDs
  → pool B's operations queue behind pool A's
  → no pool setting changes this

What actually isolates

Mechanism	Isolation provided
Separate device classes	complete — different physical devices
Separate CRUSH roots	complete — different OSDs entirely
Separate clusters	complete, at the highest cost
mClock QoS	partial — scheduler-level allocation
Client-side rate limiting	partial — bounds the noisy tenant
RBD QoS settings	per-image throttling

# device class separation
ceph osd crush rule create-replicated fast-rule default host ssd
ceph osd crush rule create-replicated bulk-rule default host hdd
ceph osd pool set rbd-fast crush_rule fast-rule
ceph osd pool set rbd-bulk crush_rule bulk-rule

This is the mechanism that works, and it works because the pools no longer share devices.

RBD-level throttling

rbd config image set rbd-vms/noisy-tenant rbd_qos_iops_limit 5000
rbd config image set rbd-vms/noisy-tenant rbd_qos_bps_limit 209715200
rbd config image set rbd-vms/noisy-tenant rbd_qos_iops_burst 10000

This bounds one image’s consumption at the client, which protects the others without requiring separate hardware. It depends on the client honouring it, so it works for trusted clients and not as a security boundary.

Diagnosing cross-tenant interference

# per-pool operation rates
ceph osd pool stats

# which pool is generating the load
ceph osd pool stats | grep -A3 'pool '

# per-image, for RBD
rbd perf image iostat --pool rbd-vms

# do the pools share OSDs?
RULE=replicated_rule
ceph osd pool get rbd-vms crush_rule
ceph osd pool get rbd-bulk crush_rule
ceph osd crush rule dump ${RULE} | grep -A3 'item_name'

If two pools use the same rule, they share OSDs and interference is expected rather than anomalous.

Designing for isolation

requirement: tenant A needs guaranteed latency
  → separate device class, or separate CRUSH root
requirement: prevent one tenant monopolising throughput
  → RBD QoS limits per image
requirement: fair sharing without guarantees
  → mClock profile, accept best-effort
requirement: hard security boundary
  → separate cluster

Quiz

Knowledge check · 4 questions

Q1. Why does mClock not isolate two client workloads from each other?

It is disabled by default, and the wpq scheduler it replaces has no notion of reservations at all Its reservations apply only to HDD-backed OSDs, since the profiles are calibrated from spinning-disk service times Its reservations are defined for operation classes — client, recovery, scrub — and both workloads fall into the client class It requires a per-pool reservation to be configured explicitly, and the default leaves every pool unreserved
Q2. RBD QoS limits can bound a noisy tenant effectively while providing no security boundary at all.

True False
Q3. Provide latency isolation for a tenant.

A tenant with a latency SLA shares OSDs with a batch analytics workload that periodically saturates the devices. Per-pool recovery_priority has been set higher for the SLA tenant with no effect.
Q4. Which mechanisms provide complete performance isolation, and which are best-effort?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Give any workload with a latency guarantee its own device class or CRUSH root; pool settings control placement and redundancy, not device capacity, so pools sharing OSDs contend regardless of configuration. Use RBD QoS limits to bound a noisy neighbour where separate hardware is not warranted.

Cross-course references

Kubernetes: resource requests do not isolate disk I/O between pods on a node either
Linux: cgroup I/O limits shape but do not guarantee shared device performance