Pool and daemon layout for RBD, CephFS, and RGW

Advanced⏱ ~18 mincephrbdradosgw-admin

What you'll learn

Lay out one pool per consumer with the right redundancy
Bind pools to device classes with CRUSH rules
Size erasure coding against the host count
Place mds, rgw, and ingress daemons deliberately

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Pools are cheap to create and expensive to merge, split, or re-class later. The layout you choose on day one is the layout you will still be running in five years, because moving a petabyte between pools is a migration project, not a command.

The pool table

Pool	Consumer	Redundancy	Device class	pg_num	Note
`.mgr`	mgr	3x	any	1	created for you
`rbd-vm`	Proxmox	3x	hdd	512	one pool, one cephx identity
`rbd-k8s`	Kubernetes	3x	ssd	256	latency-sensitive, separate class
`cephfs.acme.meta`	CephFS	3x	nvme	64	pure omap, latency-bound
`cephfs.acme.data`	CephFS	3x	hdd	512
`.rgw.root` and zone pools	RGW	3x	nvme	32	tiny, but on the write path
`acme.rgw.buckets.index`	RGW	3x	nvme	64	omap only; never sized by `ceph df`
`acme.rgw.buckets.data`	RGW	EC 4+2	hdd	512	the bulk of the capacity

Binding pools to device classes

ceph osd crush rule create-replicated on-hdd  default host hdd
ceph osd crush rule create-replicated on-ssd  default host ssd
ceph osd crush rule create-replicated on-nvme default host nvme

ceph osd pool create rbd-vm 512 512 replicated on-hdd
ceph osd pool set rbd-vm size 3
ceph osd pool set rbd-vm min_size 2
rbd pool init rbd-vm

A rule without a device class spans every OSD, which quietly puts CephFS metadata on spinning disks. The class is the whole point of buying the NVMe.

Erasure coding inside six hosts

ceph osd erasure-code-profile set ec42-hdd \
  k=4 m=2 crush-failure-domain=host crush-device-class=hdd
ceph osd pool create acme.rgw.buckets.data 512 512 erasure ec42-hdd
ceph osd pool set acme.rgw.buckets.data bulk true
ceph osd pool ls detail | grep buckets.data

Profile	Hosts needed to place	Hosts needed to self-heal	Overhead
2+2	4	5	2.0x
4+2	6	7	1.5x
6+3	9	10	1.5x
8+3	11	12	1.375x

For RBD or CephFS on an EC pool you also need overwrites, and the image still keeps its metadata on a replicated pool:

ceph osd pool set cephfs-ec-data allow_ec_overwrites true
rbd create --size 2T --data-pool rbd-ec rbd-vm/big-volume

CephFS metadata can never be erasure coded. Neither can the RGW index.

The filesystem

ceph osd pool create cephfs.acme.meta 64 64 replicated on-nvme
ceph osd pool create cephfs.acme.data 512 512 replicated on-hdd
ceph fs new acme cephfs.acme.meta cephfs.acme.data
ceph fs set acme max_mds 1
ceph fs set acme allow_standby_replay true
ceph orch apply mds acme --placement="2 node5 node6"

Daemon layout

Daemon	Count	Placement	Reason
mon	3	node1, node3, node5	odd count, spread across hosts
mgr	2	node2, node4	one active, one standby
osd	72	all six	12 per host
mds	2	node5, node6	one active, one standby-replay
rgw	3	node1-3	scales with request rate
ingress	2	node1, node2	haproxy plus keepalived on a VIP

ceph orch apply rgw acme --realm=acme --zone=acme-a --placement="3"
ceph orch ls
ceph orch ps --daemon_type mds

PG counts

ceph osd pool autoscale-status
ceph config get mon mon_target_pg_per_osd

Aim for roughly 100 PG replicas per OSD summed across all pools. With 72 OSDs that is about 7200 PG replicas, or 2400 PGs at size=3. Set bulk on the pools that will grow large so the autoscaler starts them at a full complement instead of ramping.

Quiz

Knowledge check · 4 questions

Q1. Why must the RGW bucket index pool be placed on flash?

It is pure omap, so every PUT, DELETE and listing becomes RocksDB key-value work regardless of how small the pool looks It stores the object payloads that are read most often, so serving them from flash is what keeps GET latency predictable Erasure coding requires flash devices for the parity shards, and the index pool is the only RGW pool that is erasure coded It grows faster than the data pool because every object adds an index entry, so it reaches nearfull first on spinning disks
Q2. A 4+2 erasure-coded pool on exactly six hosts recovers automatically after a host failure.

True False
Q3. Design the pool layout for a mixed RBD, CephFS and RGW cluster.

Six hosts, each with 12 HDDs and 2 NVMe devices used for block.db. Proxmox needs RBD, a small team needs CephFS, and an application needs S3 for 200 TB of archive objects.
Q4. Which pools in this design cannot be erasure coded, and why?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Bind every pool to a device class through an explicit CRUSH rule; a rule without a class spans all OSDs and puts metadata on spinning disks without warning. Choose erasure coding profiles against host count, not against overhead — k+m must fit the hosts you have, and k+m+1 is what lets the pool heal itself.

Cross-course references

Kubernetes: a StorageClass per workload is the same idea as a pool per consumer
Linux: RAID geometry chosen for capacity rather than rebuild time ages the same way