Pool quotas — bounding capacity per pool

Intermediate⏱ ~15 minceph

What you'll learn

Set and inspect pool quotas
Explain how quota enforcement works and its timing
Distinguish quotas from performance isolation
Design quotas that prevent one pool exhausting a cluster

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

A cluster shared between workloads needs a way to stop one of them consuming everything. Quotas are that mechanism, and understanding their enforcement model prevents both over-reliance and unnecessary distrust.

Setting quotas

ceph osd pool set-quota rbd-vms max_bytes 50T
ceph osd pool set-quota rgw-data max_objects 500000000
ceph osd pool get-quota rbd-vms
ceph osd pool set-quota rbd-vms max_bytes 0      # remove
ceph df detail

Both limits can be set; either being reached blocks writes.

How enforcement works

Quotas are enforced: when a pool exceeds its quota, writes to it are blocked. The nuance is timing. Usage statistics propagate from OSDs to the monitors on an interval, so enforcement is not instantaneous and a pool can overshoot its quota slightly before writes stop.

client writes  →  OSDs accumulate usage
               →  statistics reported to monitors on an interval
               →  monitors compare against quota
               →  quota exceeded flag set
               →  subsequent writes rejected

The overshoot is bounded by the reporting interval times the write rate, which for a fast-ingesting pool can be gigabytes. Set quotas with that headroom in mind rather than at the exact limit you can tolerate.

Quotas bound capacity, never performance

This is worth stating plainly because it is the most common misunderstanding. A 10 TB quota on a backup pool does not stop that pool consuming every available IOPS on the OSDs it shares.

Performance isolation requires disjoint OSDs:

ceph osd crush rule create-replicated backups_hdd default host hdd
ceph osd pool set backups crush_rule backups_hdd

So the two mechanisms address different problems and both are usually needed: quotas so one pool cannot exhaust capacity, and separate device classes or subtrees so one workload cannot exhaust device throughput.

What quotas protect against

Runaway growth. A misbehaving backup job or a bucket with no lifecycle policy filling the cluster.
Tenant overconsumption on a shared cluster.
Accidental capacity exhaustion, which on Ceph has consequences well beyond the offending pool.

That last point is why quotas matter more than they appear to. A cluster reaching full stops accepting writes across every pool, so one pool growing without limit is a cluster-wide outage risk.

Monitoring

POOL=rbd-vms
ceph df detail
ceph health detail | grep -i quota
ceph osd pool get-quota ${POOL}

POOL_NEAR_FULL and POOL_FULL health warnings fire against quotas as well as against cluster capacity, so a pool approaching its quota is visible before it stops.

Why quotas do not replace cluster-level capacity management

Quotas are per pool. The cluster-level thresholds — nearfull_ratio, backfillfull_ratio, full_ratio — are per OSD.

A cluster can have every pool comfortably within quota while individual OSDs approach full, because quotas do not account for uneven distribution, for the replication multiplier, or for capacity consumed by recovery.

Conversely, a pool at its quota may be nowhere near the cluster’s capacity, so the quota is doing its job of bounding one workload rather than protecting the cluster.

Both layers are needed and they answer different questions. Quotas answer “has this workload exceeded its allocation?”; OSD ratios answer “is the cluster running out of room?”. Monitoring only one leaves a real gap, and the OSD ratios are the ones whose breach stops everything.

Designing quotas

Sum the quotas to less than usable capacity, leaving headroom for:

recovery, which needs space to re-replicate a failed OSD,
uneven distribution across OSDs,
growth between capacity reviews.

A common allocation is to sum quotas to around 70% of usable capacity, leaving the rest as operational headroom.

Quiz

Knowledge check · 4 questions

Q1. A pool with a 10 TB quota reports 10.02 TB used before writes were blocked. Is this a fault?

No — usage statistics propagate on an interval, so a pool can overshoot slightly before writes stop Yes — the monitors hold the quota and reject writes at the exact byte, so any overshoot is a bug Yes — the quota must have been applied after the data was written, since it cannot be exceeded No — pool quotas are advisory, raising a health warning without ever blocking a write
Q2. Keeping a backup pool inside a 10 TB quota does nothing to stop it saturating the OSDs another pool depends on.

True False
Q3. Design a quota scheme for a cluster shared by four teams with 400 TB usable capacity.

400 TB usable after replication. Four teams with expected steady-state usage of 80, 60, 40, and 30 TB, growing roughly 20% annually. Capacity reviews happen twice yearly. One team previously filled a cluster with an unbounded backup job. The cluster serves production workloads across all four teams.
Q4. Explain the difference between pool quotas and cluster capacity ratios, and why both are needed.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Set quotas summing to well under usable capacity — around 70% is a reasonable allocation — leaving headroom for recovery, uneven distribution, and growth between reviews. Account for the reporting interval when setting a quota, since enforcement lags usage and a fast-ingesting pool overshoots slightly. Monitor OSD capacity ratios alongside quotas, because they answer different questions and only one of them stops the whole cluster. And remember quotas bound capacity only — throughput isolation needs disjoint OSDs.

Cross-course references

Ceph: Part LXIII (Capacity Management) for the cluster-level view.
Ceph: Part LXIV (Nearfull, Backfillfull and Full) for the OSD ratios.
Ceph: Part CIV (Multi-Tenancy in Practice) for shared-cluster design.