What happens when a tenant hits their limit

Why this matters in production

Quotas behave very differently by layer, and the RADOS pool quota is the one that fails hardest.

Behaviour at each layer

ceph osd pool get-quota tenant-acme
ceph df detail | grep -A2 tenant-acme

The pool quota is the harsh one: for RBD, a blocked write is a hung
I/O, and a guest filesystem in uninterruptible wait does not report
"quota exceeded" — it stops.

ceph -s
ceph health detail | grep -i -E 'quota|full'

Alerting before the limit

The alert that matters fires at a threshold below the limit, not at it.

- alert: TenantPoolQuotaApproaching
  expr: |
    (ceph_pool_bytes_used / on(pool_id) ceph_pool_quota_bytes) > 0.80
    and on(pool_id) ceph_pool_quota_bytes > 0
  for: 30m
  labels: { severity: warning }
  annotations:
    summary: "Pool {{ $labels.name }} is above 80% of its quota"

- alert: TenantPoolQuotaCritical
  expr: |
    (ceph_pool_bytes_used / on(pool_id) ceph_pool_quota_bytes) > 0.95
    and on(pool_id) ceph_pool_quota_bytes > 0
  for: 10m
  labels: { severity: critical }

# a manual sweep of every quota'd pool
ceph df detail --format json | python3 -c '
import sys,json
for p in json.load(sys.stdin)["pools"]:
    q = p.get("stats", {}).get("quota_bytes") or 0
    u = p.get("stats", {}).get("stored", 0)
    if q:
        print("%-24s %6.1f%%  %8.1f / %8.1f TiB" %
              (p["name"], 100.0*u/q, u/1024**4, q/1024**4))'

Responding to a tenant at their limit

ceph osd pool get-quota tenant-acme
ceph df detail | grep tenant-acme

ceph osd pool set-quota tenant-acme max_bytes $((15 * 1024**4))

Raising a pool quota takes effect immediately and unblocks hung writes,
which makes it the fastest way to end an incident — and the reason to
have an approval path agreed in advance rather than during one.

Choosing limits that fail gracefully

Prefer a limit whose breach produces an error over one that produces a
hang.

Set the pool quota as a cluster-protecting ceiling and enforce the
tenant's actual limit at the layer that returns an error.

Quiz

Production discipline

Set the tenant’s effective limit at a layer that returns an error — RBD image size, subvolume quota, or RGW quota — and keep the pool quota as a cluster-protecting ceiling above it. A pool at quota hangs RBD I/O rather than erroring, and guests cannot handle that.

Cross-course references

• Kubernetes: a limit that OOM-kills differs from one that rejects admission
• Linux: a limit that blocks is operationally worse than one that errors