Pool purpose — the unit where policy is decided

Foundation⏱ ~15 minceph

What you'll learn

Enumerate the policy decisions a pool encodes
Design a pool set for a stated workload mix
Explain what pools do and do not isolate
Document a pool design so it survives staff turnover

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Ceph has almost no global settings that decide how data is protected. Everything that matters — replication, placement, media, quotas — is a pool property, so cluster design is pool design.

What a pool decides

Decision	Setting
How many copies, or `k+m`	`size` / erasure-code profile
When writes stop	`min_size`
Which devices and which failure domain	`crush_rule`
Distribution granularity	`pg_num`
Which service owns it	`application`
Capacity ceiling	quotas
Protection against accidents	`nodelete`, `nopgchange`

ceph osd pool create rbd-vms 256 256 replicated ssd_host
ceph osd pool set rbd-vms size 3
ceph osd pool set rbd-vms min_size 2
ceph osd pool application enable rbd-vms rbd
ceph osd pool set-quota rbd-vms max_bytes 50T
ceph osd pool set rbd-vms nodelete true
ceph osd pool ls detail

What pools do and do not isolate

Do isolate: durability policy, failure domain, media class, capacity accounting, access via cephx capabilities, and namespace.

Do not isolate: device performance. Two pools sharing a CRUSH rule sit on the same OSDs and compete for the same IOPS regardless of separate quotas and sizes.

Design pools around policy, not around tenants or applications

The instinct is one pool per application. That produces twenty pools with identical settings, twenty pg_num values to manage, and PG counts per OSD that nobody planned.

Design instead around distinct policies:

nvme-metadata   size 3, nvme_host    → CephFS metadata, RGW index
ssd-general     size 3, ssd_host     → RBD for VMs
hdd-ec          k=4 m=2, hdd_rack    → RGW data, archives

Three pools serving five workloads, because those five need three policies. Separate applications within a pool using RADOS namespaces and cephx capabilities, which cost nothing.

Create a new pool when the policy differs, or when capacity accounting must be separate, or when isolation of device performance is genuinely required.

The namespace alternative

ceph auth get-or-create client.app1 \
  mon 'profile rbd' \
  osd 'profile rbd pool=rbd-vms namespace=app1'

Namespaces partition a pool’s key space, so two applications can share a pool with cephx enforcing that each sees only its own objects. This gives access isolation without multiplying pools.

Why too many pools costs more than it looks

Each pool has its own pg_num, and PG counts add up across pools. The target of 100-200 PGs per OSD is a cluster-wide total, not per pool.

Twenty pools each sized independently at 256 PGs is 5,120 PGs before replication. On a 60-OSD cluster at size 3 that is 256 PGs per OSD — above target, with the memory consequences that carries: more PG logs, larger peering structures, higher osd_memory_target pressure.

The autoscaler mitigates this by sizing pools relative to their actual data share, but it can only work with the pools that exist. A cluster with many small pools ends up with each holding the minimum viable PG count, and the sum exceeds what the OSDs should carry.

The practical rule is to count total PGs per OSD across all pools when adding a pool, not just to size the new pool sensibly on its own.

Documenting the design

For each pool, record:

the workload it serves,
size/min_size or the EC profile, and why,
the CRUSH rule and the failure it survives,
the expected capacity and growth,
who owns it.

Five lines per pool. Nothing in the cluster records any of it, and it is the first thing anyone needs during an incident.

Quiz

Knowledge check · 4 questions

Q1. Two applications need separate access control but identical durability and placement. What is the appropriate design?

One pool with RADOS namespaces and cephx capabilities scoped per namespace Two pools with identical settings and a separate cephx key for each application One pool with a separate quota per application to keep their usage apart Two pools on different CRUSH rules, since only distinct rules isolate clients
Q2. The target of 100 to 200 PGs per OSD applies to each pool independently.

True False
Q3. A team proposes one pool per application for twelve applications with three distinct durability requirements. Advise.

Cluster of 60 OSDs. Twelve applications: six need standard 3-way replication on SSD, four need the same on HDD for archival data, and two need erasure coding for large sequential data. The proposal is twelve pools so each application has its own quota and its own access key. Cluster currently has three pools.
Q4. Name what pools isolate and what they do not, and give the design implication.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Design pools around distinct policies rather than around applications or tenants — twelve applications needing three policies want three pools, with namespaces and cephx providing access separation at no PG cost. Recount total PGs per OSD across all pools whenever you add one, since the 100 to 200 target is cluster-wide. And write down five lines per pool: workload, durability, rule and the failure it survives, expected capacity, and owner. None of it is recorded in the cluster.

Cross-course references

Ceph: Part XVIII (Placement Groups) for the PG budget.
Ceph: Part XXXII (Least Privilege Capabilities) for namespace scoping.
Ceph: Part XV (CRUSH Maps and Rules) for the placement half.