File layouts: how a CephFS file maps to objects

Advanced⏱ ~17 mingetfattrsetfattr

What you'll learn

Read and interpret a file layout
Set layouts on directories for inheritance
Choose striping parameters for a workload
Place a subtree on a different data pool

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Layouts are the mechanism for tiering and for tuning within a single namespace, and they are set through extended attributes rather than through Ceph commands — which is why they are frequently overlooked entirely.

Reading a layout

getfattr -n ceph.file.layout /mnt/cephfs/data/large.bin
# ceph.file.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs-data"

getfattr -n ceph.dir.layout /mnt/cephfs/data

Attribute	Meaning
`stripe_unit`	bytes written to one object before moving to the next
`stripe_count`	how many objects are striped across before wrapping
`object_size`	maximum size of each object
`pool`	which data pool holds the file

The default — stripe_unit equal to object_size with stripe_count 1 — means simple sequential mapping: bytes 0–4 MiB in object 0, 4–8 MiB in object 1.

Setting layouts

Layouts are inherited from the directory at file creation and cannot be changed on a file that already has data:

mkdir /mnt/cephfs/bigfiles
setfattr -n ceph.dir.layout.object_size -v 16777216 /mnt/cephfs/bigfiles
setfattr -n ceph.dir.layout.stripe_unit  -v 16777216 /mnt/cephfs/bigfiles

# files created here now use 16 MiB objects

# whole layout in one attribute
setfattr -n ceph.dir.layout \
  -v "stripe_unit=1048576 stripe_count=8 object_size=4194304 pool=cephfs-fast" \
  /mnt/cephfs/parallel

Striping for parallelism

The same reasoning as RBD: a single-threaded sequential writer touches one object at a time and is capped at one OSD’s throughput. Striping spreads it:

stripe_unit=1M, stripe_count=8, object_size=4M
  bytes 0–1M     → object 0
  bytes 1M–2M    → object 1
  ...
  bytes 7M–8M    → object 7
  bytes 8M–9M    → object 0 (second stripe within it)

Worth doing for large single-stream writes; unnecessary for concurrent workloads, which spread naturally.

Tiering with pools

ceph fs add_data_pool cephfs cephfs-archive
setfattr -n ceph.dir.layout.pool -v cephfs-archive /mnt/cephfs/archive

Files created under /archive land on the EC archive pool while the rest of the filesystem stays on replicated storage — one namespace, two storage tiers, transparent to users.

Existing files do not move. Placing existing data on a new pool means copying it:

cp -a /mnt/cephfs/old/data /mnt/cephfs/archive/data && rm -rf /mnt/cephfs/old/data

Quiz

Knowledge check · 4 questions

Q1. You set `ceph.dir.layout.pool` on a directory containing existing files. What happens to those files?

They migrate to the new pool in the background, at the configured recovery rate They become inaccessible, since the client now computes their addresses in the new pool They keep their original layout and pool — only newly created files use the new one They are copied to the new pool on next access and removed from the old one afterwards
Q2. A file's data pool is recorded in its layout so the client can compute object addresses without further MDS interaction.

True False
Q3. Introduce tiering into an existing CephFS deployment.

A 200 TB CephFS filesystem on replicated storage holds a mix of active project data and old completed projects. The team wants completed projects on an EC pool to reclaim capacity, without changing how users access the filesystem.
Q4. Why is a file's layout immutable once it contains data?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Set directory layouts before data lands in them; retrofitting a layout means copying every affected file, which for a large subtree is a migration project. Document which subtrees use non-default layouts, since the setting is invisible in ordinary directory listings and easily lost during reorganisation.

Cross-course references

Kubernetes: StorageClass selection at PVC creation has the same at-creation-only property
Linux: filesystem allocation policies set at mkfs time are similarly permanent