PGs per pool — managing the cluster-wide total

Intermediate⏱ ~15 minceph

What you'll learn

Compute the cluster-wide PG total across pools
Allocate the budget across pools by data share
Monitor PGs per OSD as a standing metric
Rebalance the allocation when pool shares change

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Pools are sized one at a time and the budget is shared. That mismatch is how clusters end up with three times the PGs per OSD that anyone intended.

Computing the total

ceph osd pool ls detail | grep -oP 'pg_num \K[0-9]+'
ceph pg stat
ceph osd df tree | awk 'NR>1 && $NF ~ /^osd/ {print}' | head

A direct calculation:

total=0
for p in $(ceph osd pool ls); do
  pg=$(ceph osd pool get $p pg_num -f json | jq -r .pg_num)
  sz=$(ceph osd pool get $p size -f json 2>/dev/null | jq -r .size)
  sz=${sz:-3}
  total=$(( total + pg * sz ))
done
osds=$(ceph osd ls | wc -l)
echo "$(( total / osds )) PGs per OSD"

Under 100 means room to grow; over 200 means investigate; over 300 means memory and peering pressure are likely.

ceph df

Read the %USED column per pool, and allocate the budget in proportion. A pool holding 5% of the data does not need 25% of the PGs, and giving it that means another pool is short.

Small pools should get small PG counts, and often the minimum

The instinct to give every pool a healthy PG count is what inflates the total. A metadata pool holding 20 GB does not need 512 PGs; it needs enough to spread across the OSDs it uses.

ceph osd pool autoscale-status

The autoscaler handles this correctly, sizing by actual share. On a hand-managed cluster, the check is to compare each pool’s PG share against its data share:

ceph df --format json | jq -r '.pools[] | "\(.name) \(.stats.percent_used)"'

A pool with 2% of the data and 20% of the PGs is consuming budget that the pool holding 60% of the data needs.

Monitoring it

Add PGs per OSD to standing metrics. It changes slowly and it is invisible until it causes a problem, which is the profile of a metric worth graphing.

ceph pg stat
ceph health detail | grep -iE 'too many|too few'

Ceph raises TOO_MANY_PGS and TOO_FEW_PGS health warnings against configurable thresholds, which is a useful backstop but fires only at the extremes.

Why adding OSDs raises the budget and adding pools spends it

The budget is OSD count times 100 to 200. Adding OSDs raises it proportionally, so a cluster growing from 60 to 96 OSDs can carry 60% more PGs without changing anything.

Adding a pool spends from the same budget. A new pool at 512 PGs with size 3 consumes 1,536 PG-replicas, which on 60 OSDs is 26 per OSD — a quarter of the whole budget at the low end of the target range.

The practical consequence is that pool creation on a mature cluster should include a budget check, not just a sensible pg_num for the new pool. Three new pools added over a year, each individually reasonable, can push a cluster from 120 to 200 PGs per OSD without anyone making a decision about it.

# before creating a pool
ceph pg stat
ceph osd ls | wc -l

Rebalancing the allocation

When pool shares change — a pool grows tenfold, another is decommissioned — the allocation should follow:

GROWING_POOL=rbd-vms
RECOMMENDED=recommended
ceph osd pool set ${GROWING_POOL} pg_autoscale_mode warn
ceph osd pool autoscale-status
ceph osd pool set ${GROWING_POOL} pg_num ${RECOMMENDED}

Raise the growing pool rather than shrinking the shrinking one, since raising is the cheap direction and the total may still be within budget.

Quiz

Knowledge check · 4 questions

Q1. A 60-OSD cluster at 120 PGs per OSD adds a new pool with pg_num 512 and size 3. What is the effect on the per-OSD count?

Negligible, since one small pool among seven changes the total very little It adds about 9 PGs per OSD, since size divides the count rather than multiplying it It has no effect until the pool receives data and its PGs are actually created It adds about 26 PGs per OSD, roughly a quarter of the low-end budget
Q2. A pool holding 2% of a cluster data should be allocated roughly 2% of the PG budget.

True False
Q3. A cluster at 210 PGs per OSD is asked to host a new application needing its own pool. Advise.

60 OSDs, seven existing pools totalling 210 PGs per OSD at size 3. OSDs run at 5 GiB RSS against a 4 GiB target on hosts with modest headroom. The new application needs about 500 GB of storage with standard 3-way replication on the existing SSD tier, and asks for its own pool for capacity accounting.
Q4. Explain why adding OSDs raises the PG budget while adding pools spends it.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Track PGs per OSD as a standing metric — it changes slowly, is invisible until it causes memory and peering problems, and is exactly the shape of thing worth graphing. Check the budget before creating a pool rather than only choosing a sensible pg_num for it, since individually reasonable additions accumulate. Allocate in proportion to data share, giving small pools small counts. And when shares change, raise the growing pool rather than shrinking the shrinking one.

Cross-course references

Ceph: Part XXI (PG Autoscale) for automating the allocation.
Ceph: Part XVII (Pools) for deciding how many pools to have.
Ceph: Part XI (OSD Architecture) for the memory this consumes.