Pools — the unit of policy

Foundation⏱ ~15 minceph

What you'll learn

List the policy decisions a pool encodes
Create and configure a pool for a stated requirement
Explain how pools share and compete for the same OSDs
Apply quotas and application tags correctly

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Ceph has very few global settings that matter for data. Almost everything that decides how data is protected, where it lives, and who can reach it is a property of a pool. Designing a cluster largely means designing its pools.

What a pool encodes

Setting	Decides
`size` / `min_size`	how many copies, and how few before writes block
erasure code profile	k+m instead of replication
`crush_rule`	which devices and which failure domain
`pg_num` / `pgp_num`	how finely data is distributed
`application`	which service owns it (rbd, cephfs, rgw)
quotas	maximum bytes or objects
`nodelete`, `nopgchange`	protective flags

Creating one:

ceph osd pool create rbd-vms 256 256 replicated ssd_rule
ceph osd pool set rbd-vms size 3
ceph osd pool set rbd-vms min_size 2
ceph osd pool application enable rbd-vms rbd
ceph osd pool set-quota rbd-vms max_bytes 50T

This is the property most often missed. Two pools with the same CRUSH rule sit on the same OSDs and compete for the same device IOPS.

So a backup pool being written hard slows the VM pool, even though they are separate pools with separate quotas and separate sizes. Quotas limit capacity, not performance.

Real isolation requires separate devices:

ceph osd crush rule create-replicated nvme_rule default host nvme
ceph osd crush rule create-replicated hdd_rule default host hdd
ceph osd pool set rbd-vms crush_rule nvme_rule
ceph osd pool set backups crush_rule hdd_rule

Now the pools sit on disjoint sets of OSDs and genuinely cannot interfere.

Protective flags

ceph osd pool set rbd-vms nodelete true
ceph osd pool set rbd-vms nopgchange true
ceph config set mon mon_allow_pool_delete false

mon_allow_pool_delete false is worth setting on every production cluster. Deleting a pool destroys every object in it immediately, with no undo and no recycle bin, and it is a single command.

Why pg_num changes cause data movement and pgp_num exists

pg_num is how many placement groups a pool has. pgp_num is how many of them are actually used for placement.

Splitting PGs by raising pg_num divides existing PGs into smaller ones, which is metadata work. Raising pgp_num then changes where those PGs map, which is data movement. Separating the two lets an operator split first and then move gradually.

In practice the autoscaler manages both and manual tuning is rarely needed. What remains important is the direction: raising pg_num is routine, while lowering it merges PGs and moves substantial data. Neither is free, and both should happen during a window with recovery throttled — which is the main argument for getting the initial value approximately right rather than relying on later correction.

Sizing pg_num

The rough target is 100-200 PGs per OSD across all pools combined:

pg_num ≈ (OSDs x 100) / size,  rounded to a power of two

For 60 OSDs at size 3: 60 × 100 / 3 = 2000, round to 2048 — but that is for a pool holding all the data. With several pools, divide according to expected capacity share. The autoscaler does this arithmetic automatically and is the recommended default.

Quiz

Knowledge check · 4 questions

Q1. A backup pool and a VM pool have separate quotas and separate sizes, yet heavy backup writes slow the VMs. Why?

The quotas are set too high, so backup writes consume the capacity headroom the VMs rely on Both pools use the same CRUSH rule, so they sit on the same OSDs and share device IOPS The backup pool holds too many placement groups, starving the VM pool of its share of them Ceph prioritises larger writes in the OSD op queue, so backup traffic preempts the VM I/O
Q2. Setting mon_allow_pool_delete to false is a worthwhile precaution on a production cluster.

True False
Q3. A cluster has 14 pools accumulated over four years. Three have no application tag and nobody knows what two of them contain. Plan a cleanup.

Production cluster, 96 OSDs. ceph df shows two untagged pools holding 40 GB and 2 TB respectively. POOL_APP_NOT_ENABLED has been muted for two years. No documentation exists. The team wants to reclaim capacity and reduce PG count, and someone has proposed deleting both pools.
Q4. Explain the difference between pg_num and pgp_num and why raising pg_num is safer than lowering it.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Tag every pool with its application, and never mute POOL_APP_NOT_ENABLED — a silenced warning is how clusters end up with pools nobody can identify. Set mon_allow_pool_delete false on production, since pool deletion is immediate and irreversible. And remember that quotas bound capacity, not performance: two pools that must not interfere need disjoint OSDs through separate device classes and CRUSH rules, which is a design decision rather than a consequence of being separate pools.

Cross-course references

Ceph: Part XVI (Device Classes) for building disjoint tiers.
Ceph: Part XVIII (Placement Groups) for pg_num sizing.
Ceph: Part XXV (Erasure Coding Fundamentals) for the EC pool variant.