Creating a CephFS filesystem correctly the first time

Why this matters in production

ceph fs volume create produces a working filesystem in one command, with pools sized by the autoscaler and placed on whatever CRUSH default applies. That is fine for a test and rarely right for production, and several of the resulting choices are awkward to revisit.

The explicit form

# metadata pool: small, flash, generous PG floor
ceph osd pool create cephfs-meta 128 128 replicated meta-nvme
ceph osd pool set cephfs-meta size 3
ceph osd pool set cephfs-meta pg_num_min 128
ceph osd pool set cephfs-meta pg_autoscale_bias 4

# data pool
ceph osd pool create cephfs-data 1024 1024 replicated by-host
ceph osd pool set cephfs-data size 3
ceph osd pool set cephfs-data bulk true

# the filesystem
ceph fs new cephfs cephfs-meta cephfs-data
ceph fs status cephfs

The metadata pool decisions — flash placement and a PG floor — are the two that matter most and the two the convenience command will not make for you.

Settings worth setting at creation

ceph fs set cephfs max_mds 1
ceph fs set cephfs standby_count_wanted 1
ceph fs set cephfs allow_standby_replay true
ceph fs set cephfs max_file_size 1099511627776      # 1 TiB

max_file_size defaults to 1 TiB and caps how large any single file may be. It can be raised later, but a workload that needs larger files fails in a confusing way until it is.

Verifying

ceph fs status cephfs
ceph fs get cephfs
ceph fs dump
ceph osd pool ls detail | grep -A5 cephfs

Check the pools are on the intended CRUSH rules, that max_mds and standby settings are as intended, and that MDS daemons have been assigned.

Changeable versus awkward

The original data pool cannot be removed from a filesystem, so a pool chosen carelessly at creation is permanent. Additional data pools can be added and used via layouts, but the first one stays.

Multiple filesystems

ceph fs volume create scratch
ceph fs ls

Separate filesystems have separate MDS daemons and separate metadata pools — genuine resource isolation, at the cost of more daemons. Use them where one workload’s metadata load would otherwise disturb another.

Quiz

Production discipline

Create CephFS pools explicitly rather than through the convenience command for any production filesystem; the metadata pool’s device class and PG floor are the two decisions it will not make correctly and the two that matter most. Choose the original data pool deliberately, since it is the one part of the configuration that is genuinely permanent.

Cross-course references

• Kubernetes: default StorageClass selection has the same convenient-but-often-wrong property
• Linux: mkfs defaults are similarly reasonable and similarly worth overriding for production