PG count tuning — the numbers and what they cost

Intermediate⏱ ~16 minceph

What you'll learn

Compute an appropriate pg_num for a pool
Balance PG counts across pools within a cluster budget
Change pg_num safely on a live pool
Recognise when PG count is the cause of a problem

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

PG count affects distribution evenness, memory consumption, peering duration, recovery parallelism, and scrub granularity. It is one number with five consequences.

The arithmetic

target total PGs = osd_count x 100        (to 200 at the top end)

pg_num per pool  = target_total x pool_share / size

Worked example: 60 OSDs, target 100 per OSD, three pools holding 50%, 30%, and 20% of the data at size 3:

target total = 60 x 100 = 6000 PG-replicas
pool A: 6000 x 0.50 / 3 = 1000  → round to 1024
pool B: 6000 x 0.30 / 3 =  600  → round to  512
pool C: 6000 x 0.20 / 3 =  400  → round to  512
total PG-replicas = (1024 + 512 + 512) x 3 = 6144  → 102 per OSD

Round to powers of two, then check the total.

ceph pg stat
ceph osd df tree | awk 'NR>1 {s+=$(NF-1); n++} END {print s/n, "PGs per OSD"}'
ceph osd pool autoscale-status

Changing pg_num

ceph config set osd osd_max_backfills 1
ceph osd pool set rbd-vms pg_num 2048
ceph osd pool get rbd-vms pg_num          # climbs toward the target
ceph -s

Ceph applies the change incrementally, so a large increase on a large pool takes hours to days. The climbing value is progress, not a stall.

Lowering pg_num is the expensive direction and rarely the right answer

Raising splits PGs cleanly: one becomes two, objects divide by a hash bit, and movement is proportionate.

Lowering merges: two PGs combine, their contents must be brought together, and substantially more data moves.

If a pool has too many PGs, the usual reasons are that pg_num was set for a larger cluster, or that many pools were each sized generously. Before merging, consider:

Is the cluster going to grow? More OSDs raise the budget, and the count may be right in six months.
Can pools be consolidated instead? Fewer pools with the same total PGs is a better shape.
Is the count actually causing a problem? Memory pressure and slow peering are the symptoms; if neither is present, leave it.

Merge only when a real symptom justifies the movement.

Symptoms of a wrong count

Symptom	Likely count problem
uneven OSD utilisation the balancer cannot fix	too few PGs
very long individual scrubs or backfills	too few, so PGs are large
high OSD memory, OOM during peering	too many
slow peering after every map change	too many
poor recovery parallelism	too few

Why powers of two matter more than the exact target

The target of 100-200 PGs per OSD is a broad range, and a pool at 90 or at 210 behaves essentially the same.

Powers of two are not similarly flexible. Because the PG id is the object hash masked by pg_num - 1, a non-power value means the mask does not take clean low bits and some PGs receive from more hash values than others. The distribution is uneven regardless of how well CRUSH performs the second stage.

Ceph accepts non-power values and the autoscaler recommends powers of two. If you find a pool at 1000 rather than 1024, that is worth correcting on the next planned change — not urgently, since the unevenness is modest, but there is no reason to keep it.

The practical guidance is therefore: get within the range, and land on a power of two. Precision within the range buys nothing.

The recommended approach

Let the autoscaler compute it, in warn mode on an established cluster, and apply its recommendations deliberately. Understand the arithmetic so you can judge whether a recommendation makes sense — particularly for pools about to grow, where target_size_ratio is the right way to inform it.

Quiz

Knowledge check · 4 questions

Q1. A 60-OSD cluster at size 3 has three pools holding 50%, 30%, and 20% of the data. What total pg_num across the pools targets 100 PGs per OSD?

About 6000, split by pool share across the three pools About 2000, since the replication factor divides the target About 18000, since each PG is replicated three times over About 600, allocating ten PGs to each OSD in every pool
Q2. A pool with too many PGs should be merged down promptly to reach the target range.

True False
Q3. A cluster shows uneven OSD utilisation that the balancer cannot correct, and long individual deep scrubs. Diagnose.

48 OSDs, one pool with pg_num 256 at size 3, holding 300 TB. That gives 16 PGs per OSD. ceph osd df shows utilisation from 51% to 79%. The balancer is enabled in upmap mode and reports no further improvements. Individual deep scrubs take many hours and produce noticeable latency while running.
Q4. List the five things PG count affects and name the symptom of too few.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Compute the budget as OSDs times 100 to 200 PG-replicas, divide by size, and split by pool share — the distinction between PG-replicas and pg_num is where the arithmetic usually goes wrong. Land on powers of two, since the hash mask depends on it, and do not chase precision within the range. Raise pg_num deliberately and choose the target carefully, because raising is cheap and merging back is not.

Cross-course references

Ceph: Part XXI (PG Autoscale) for automating this.
Ceph: Part XI (OSD Architecture) for the memory consequences.
Ceph: Part LXI (Scrubbing) for scrub duration and PG size.