Serving multiple consumers from one cluster

Advanced⏱ ~18 minceph

What you'll learn

Isolate consumers appropriately
Attribute usage and load per consumer
Prevent one consumer affecting another
Onboard a new consumer safely

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

One cluster serving several teams is efficient and creates the possibility of one consumer harming another. The isolation mechanisms are specific.

The isolation layers

Layer	Mechanism	Isolation provided
Authentication	separate cephx users	access control
Authorisation	per-pool capabilities	a consumer sees only their pools
Capacity	per-pool quotas	one cannot consume the cluster
Namespace	RBD namespaces, CephFS subvolumes	within a shared pool
Performance	separate device classes or CRUSH roots	genuine isolation
Rate	RBD QoS limits	bounds a noisy consumer

# a consumer's user, scoped to their pools
ceph auth get-or-create client.team-a \
  mon 'profile rbd' \
  osd 'profile rbd pool=team-a-vms' \
  mgr 'profile rbd pool=team-a-vms'

The capability restricts the user to that pool; a key scoped correctly cannot read or write another team’s data.

Capacity attribution and limits

ceph osd pool set-quota team-a-vms max_bytes 50T
ceph osd pool get-quota team-a-vms
ceph df detail

# per-consumer usage, for chargeback or reporting
sum by (name) (ceph_pool_stored)
  * on(pool_id) group_left(name) ceph_pool_metadata

Quotas do two things: they prevent one consumer consuming the cluster, and they make the consumer’s own limit visible to them rather than the cluster’s.

Performance isolation

# separate device classes give genuine isolation
ceph osd crush rule create-replicated team-a-rule default host nvme
ceph osd pool set team-a-vms crush_rule team-a-rule

# rate limiting bounds a noisy consumer without separate hardware
rbd config pool set team-b-vms rbd_qos_iops_limit 20000
rbd config pool set team-b-vms rbd_qos_bps_limit 2147483648

Only separate devices provide a guarantee; rate limiting is best-effort and depends on clients honouring it.

Onboarding a consumer

1. capacity requirement gathered and checked against available
2. pool created with an appropriate CRUSH rule and durability policy
3. quota set to the stated requirement plus margin
4. cephx user created, scoped to that pool
5. rate limits applied if the pool shares devices
6. monitoring and alerting configured, routed to the consumer
7. service commitments stated
8. consumer-facing dashboard created
9. the consumer connects and verifies

# the mechanical part
ceph osd pool create team-c-vms 128 128 replicated team-c-rule
ceph osd pool application enable team-c-vms rbd
rbd pool init team-c-vms
ceph osd pool set-quota team-c-vms max_bytes 30T
ceph auth get-or-create client.team-c \
  mon 'profile rbd' osd 'profile rbd pool=team-c-vms' mgr 'profile rbd pool=team-c-vms'

Preventing cross-consumer harm

Risk	Mitigation
One consumer fills the cluster	quotas on every pool
One consumer saturates I/O	rate limits, or separate device classes
One consumer’s recovery affects others	recovery throttles, priority per pool
A consumer accessing another’s data	scoped cephx capabilities
A consumer’s misconfiguration	they cannot change cluster settings

# verify a consumer's key cannot exceed its scope
ceph auth get client.team-a

Quiz

Knowledge check · 4 questions

Q1. Why does a per-pool quota change what a consumer sees as available capacity?

It enables compression for the pool, so the reported figure reflects what the consumer's data will occupy once compressed rather than the raw bytes written It reserves the bytes physically on the OSDs the moment it is set, so they are removed from what every other pool can be offered It changes the durability policy, since a quota is applied after replication and the consumer therefore sees post-replication capacity MAX AVAIL becomes bounded by the quota, so the consumer sees their own limit rather than the whole cluster's remaining capacity that others also count on
Q2. RBD QoS rate limits provide a guarantee of performance isolation between consumers.

True False
Q3. Onboard a second consumer onto a single-consumer cluster.

A dedicated Ceph cluster has served one Proxmox environment for two years with no quota on its pool. A second team is being onboarded.
Q4. Which isolation mechanisms provide a guarantee and which are best-effort?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Set a quota on every consumer pool including the first — the unquotaed pool consumes whatever the others leave, and it is usually the oldest and largest. Give any consumer with a latency commitment its own device class; rate limits shape behaviour but do not guarantee it.

Cross-course references

Kubernetes: namespace quotas and network policies serve the same multi-tenant role
Linux: shared services need per-consumer limits from the first consumer onward