Why placement groups exist

Foundation⏱ ~15 minceph

What you'll learn

Explain the scaling problem PGs solve
Describe what an OSD tracks per PG rather than per object
Relate PG count to memory, peering, and recovery cost
Justify the 100-200 PGs per OSD guidance

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Placement groups are the one Ceph concept with no equivalent in single-server storage, and almost every capacity, memory, and recovery consideration touches them.

The problem they solve

Without PGs, the cluster would need to track, for each object, which OSDs hold it. On a petabyte of 4 MiB objects that is 260 million entries — a table growing with data volume, replicated for durability, and consulted constantly.

With PGs there are two mappings:

object  →  PG      cheap hash, no state
PG      →  OSDs    CRUSH computation over a small map

Neither stores per-object state. The number of PGs is chosen by the operator and stays constant as data grows, so the tracking cost is bounded by cluster size rather than by object count.

What an OSD tracks per PG

membership: which OSDs are in the acting and up sets,
a PG log of recent operations,
peering state and history,
scrub scheduling and status,
statistics.

None of this is per object. An OSD holding 200 PGs maintains 200 of each structure whether those PGs hold a thousand objects or a million.

PG count is a cluster-wide budget, not a per-pool setting

The 100-200 PGs per OSD guidance is the sum across every pool, multiplied by replication.

ceph pg stat
ceph osd df tree | awk 'NR>1 {print $1, $(NF-1)}'   # PGs per OSD

total PGs per OSD = sum over pools of (pg_num x size) / osd_count

A cluster with eight pools each at 512 PGs and size 3 on 60 OSDs carries 205 PGs per OSD — at the top of the range from pools that each looked reasonable alone.

Recount the total whenever a pool is added, and prefer fewer pools with more PGs each over many pools with the minimum.

What too few and too many cost

Too few (under about 50 per OSD):

uneven data distribution, since CRUSH has fewer samples,
coarse balancing granularity, so the balancer cannot even things out,
large individual PGs, making each scrub, backfill, and split a long operation,
limited recovery parallelism.

Too many (over about 300 per OSD):

more PG logs, peering structures, and per-PG state in memory,
longer and more memory-hungry peering after every map change,
more monitor and manager work tracking states.

Why the peering cost of PG count is the one that bites

Steady-state memory from PGs is significant and predictable. The problem is peering.

After any map change, every affected OSD peers every affected PG, building temporary state per PG. With 200 PGs that is manageable. With 600 it is three times the transient allocation, on every OSD, at the same moment.

The failure mode is an OOM cascade: peering exhausts memory, the OOM killer terminates an OSD, that produces a map change, which triggers peering on every remaining OSD, which consumes more memory.

This is why PG count is a sizing input for host memory rather than only a distribution parameter, and why a cluster running comfortably at 500 PGs per OSD in steady state can fail during its first large recovery.

ceph daemon osd.12 dump_mempools | jq '.mempool.by_pool.osd_pglog.bytes'

Computing a target

pg_num per pool ≈ (osd_count x 100) / size / pool_share

For 60 OSDs, size 3, one pool holding everything: 60 × 100 / 3 = 2000, round to 2048. Split across pools by expected data share.

The autoscaler does this arithmetic continuously and is the recommended default; understanding it matters for judging its recommendations and for the cases it does not model.

Quiz

Knowledge check · 4 questions

Q1. What scaling problem do placement groups solve?

They reduce the number of OSDs each write must reach, since only the PG primary has to acknowledge it They let an object larger than a single device be split across several OSDs transparently They avoid per-object placement tracking, so state scales with cluster size rather than object count They give the write ordering guarantee replication needs, since a PG serialises its own operations
Q2. The 100 to 200 PGs per OSD guidance applies to the sum across all pools multiplied by replication.

True False
Q3. A cluster with 430 PGs per OSD runs fine in steady state but OSDs were OOM-killed during the last recovery. Explain the connection.

96 OSDs, 64 GB RAM per host with 12 OSDs each, osd_memory_target 4 GiB. Eleven pools sized independently over four years give 430 PGs per OSD. Steady-state memory usage is around 5.5 GiB per OSD. During a recent host failure, peering pushed several OSDs past available memory and the OOM killer terminated them, which triggered further peering.
Q4. List what an OSD tracks per placement group and explain why that is not per object.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Treat PG count as a cluster-wide budget summed across pools and multiplied by replication, and recount it whenever a pool is added. Size host memory against the peering peak rather than steady state, because peering allocates per-PG state on every OSD simultaneously and that is where an over-PG’d cluster fails. And prefer fewer pools with more PGs each to many pools each carrying the minimum, since the minimums add up faster than anyone expects.

Cross-course references

Ceph: Part XI (OSD Architecture) for the memory consequences.
Ceph: Part XX (PG Peering) for what peering allocates.
Ceph: Part XXI (PG Autoscale) for automating the arithmetic.