Pool-level blocking conditions

Why this matters in production

One pool blocked while everything else works points at pool-level configuration, and the sweep for it is short and specific.

The conditions

ceph osd pool ls detail | grep -A2 "^pool.*'rbd-vms'"
ceph osd pool get-quota rbd-vms

Quotas

ceph osd pool get-quota rbd-vms
# quotas for pool 'rbd-vms':
#   max objects: N/A
#   max bytes  : 50 TiB  (current num bytes: 53687091200000)

# raise, or remove
ceph osd pool set-quota rbd-vms max_bytes 80T
ceph osd pool set-quota rbd-vms max_bytes 0      # 0 removes the limit

Quota enforcement depends on periodically-updated statistics, so a burst can exceed the limit before it takes effect — which is why the current figure may exceed the configured maximum.

CRUSH rule placement failure

RULE=replicated_rule
ceph osd pool get rbd-vms crush_rule
ceph osd crush rule dump ${RULE}
ceph pg dump pgs | awk '$10 !~ /active/ {print $1, $10}' | head

A rule requiring more failure domains than exist leaves PGs unable to peer:

size=3, failure_domain=rack, 2 racks available
  → CRUSH cannot select a third rack
  → PGs stay undersized or inactive

# how many of the failure domain exist?
ceph osd tree | grep -c 'rack '
ceph osd pool get rbd-vms size

The protection flags

ceph osd pool set rbd-vms nodelete true
ceph osd pool set rbd-vms nosizechange true
ceph osd pool set rbd-vms nopgchange true

These prevent accidental changes and produce confusing failures when forgotten — an operator attempting to raise size receives a refusal with no obvious cause.

ceph osd pool ls detail | grep -oE 'no(delete|sizechange|pgchange)'

Preventing recurrence

- alert: CephPoolQuotaApproaching
  expr: ceph_pool_stored / ceph_pool_quota_bytes > 0.85
  for: 1h
  labels: { severity: ticket }

Quiz

Production discipline

Check ceph osd pool get-quota first when one pool is blocked and the cluster is healthy — it is the most common cause and takes one command. Verify the failure domain count against the pool size before setting any CRUSH rule; a rule that cannot place is permanent until the topology or the rule changes.

Cross-course references

• Kubernetes: a namespace-scoped limit blocking one workload while the cluster is idle
• Linux: per-user quotas produce identical single-tenant blocking