CephXL · CephFS ArchitectureCephFS Architecture
The metadata pool and the data pools
What you'll learn
- Explain the role and contents of each pool
- Size and place the metadata pool appropriately
- Add and use additional data pools
- Recognise metadata pool problems from their symptoms
Prerequisites
None — start here.
Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18
Why this matters in production
The metadata pool holds almost no bytes and determines almost all of the filesystem’s responsiveness. Treating it as a minor pool — a few PGs, on whatever devices are available — is one of the more consequential CephFS mistakes and one of the easiest to avoid.
The two pools
ceph osd pool create cephfs-meta 64 64 replicated
ceph osd pool create cephfs-data 512 512 replicated
ceph fs new cephfs cephfs-meta cephfs-data
ceph fs status cephfs| Metadata pool | Data pool | |
|---|---|---|
| Contents | inodes, dentries, MDS journal | file contents |
| Size | tiny — often under 1% of data | everything |
| Access pattern | small random, omap-heavy | matches the workload |
| Latency sensitivity | extreme | workload-dependent |
| Erasure coding | not possible — omap | possible |
| Device class | flash strongly preferred | as appropriate |
Placing metadata on flash
ceph osd crush rule create-replicated meta-nvme default host nvme
ceph osd pool set cephfs-meta crush_rule meta-nvme
ceph osd pool set cephfs-meta size 3Every namespace operation — every stat, every open, every directory
listing — involves metadata pool I/O. Placing it on NVMe while data lives
on HDD is the single highest-return CephFS configuration decision, and the
pool is small enough that the flash cost is modest.
Give it a PG floor too, since its size does not reflect its need for parallelism:
ceph osd pool set cephfs-meta pg_num_min 128
ceph osd pool set cephfs-meta pg_autoscale_bias 4Additional data pools
ceph fs add_data_pool cephfs cephfs-archive
setfattr -n ceph.dir.layout.pool -v cephfs-archive /mnt/cephfs/archiveA directory can be assigned a different data pool, so cold data lands on an EC pool while active data stays on replicated flash — within one namespace. Files inherit the layout of the directory they are created in; existing files do not move.
getfattr -n ceph.dir.layout /mnt/cephfs/archive
getfattr -n ceph.file.layout /mnt/cephfs/archive/old.tarMetadata pool symptoms
| Symptom | Points at |
|---|---|
ls slow, transfers fine | metadata pool latency |
| File creation slow | metadata pool or MDS |
| MDS slow requests | metadata pool latency |
| Everything slow | data pool or cluster-wide |
Quiz
Knowledge check · 4 questions
Q1. Why can a CephFS metadata pool not be erasure coded?
Q2. Placing the CephFS metadata pool on flash while data stays on HDD is usually a high-return configuration choice.
Q3. Improve a CephFS deployment that feels slow for interactive use.
Users report that `ls` and file operations in a CephFS deployment take seconds while large file transfers achieve full expected throughput. All pools are on HDD. The metadata pool holds 40 GB against 300 TB of data.
Q4. Why does metadata pool latency amplify into filesystem-wide slowness?
Passing score: 75%. Answers are checked in this browser.
Production discipline
Place the metadata pool on the fastest device class available and give it an explicit PG floor — its size does not reflect its need for parallelism and the autoscaler will otherwise starve it. Monitor metadata pool latency separately from data pool latency, since it amplifies through the MDS into everything users experience.
Cross-course references
- Kubernetes: etcd on fast storage matters far beyond its size for the same amplification reason
- Linux: separating a filesystem journal onto a fast device follows the identical logic