The CephFS split path: metadata through the MDS, data direct

Intermediate⏱ ~17 minceph

What you'll learn

Trace a file open and read through both paths
Explain why data bypasses the MDS
Predict which operations are MDS-bound
Use the split to structure a performance investigation

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

CephFS performance problems divide cleanly into two categories that share almost no diagnostics. A workload creating millions of small files stresses the MDS; a workload streaming large files stresses the OSDs. Knowing which path a complaint lives on eliminates half the possible causes immediately.

The two paths

graph TD
    C[Client] -->|open, stat, readdir, rename| M[MDS]
    M -->|inode, layout, capability| C
    C -->|read and write file data| O[OSDs]
    M -->|journal and metadata objects| O

The client asks the MDS to resolve a path, and receives the inode number, the file layout, and a capability. From that point it computes object names itself and reads and writes directly against the OSDs. The MDS never touches file contents.

A worked open-and-read

1. client → MDS:  open("/data/reports/q3.csv", O_RDONLY)
2. MDS:           traverse the namespace, check permissions
3. MDS → client:  inode 0x10000003a2f, layout, read capability
4. client:        compute object names from inode and layout
                  → 10000003a2f.00000000, 10000003a2f.00000001, ...
5. client → OSDs: read those objects directly

Step 5 involves no MDS at all. A client reading a large file it already has open generates zero MDS traffic.

Which operations hit which path

Operation	Path
`open`, `create`, `unlink`	MDS
`stat`, `readdir`, `rename`	MDS
`chmod`, `chown`, `setxattr`	MDS
`read`, `write` on an open file	OSDs directly
`fsync`	OSDs, plus MDS for size updates
`ls -l` on a large directory	MDS, heavily

The pattern: anything touching the namespace goes to the MDS; anything touching bytes goes to the OSDs.

Structuring an investigation

# is this a metadata or a data problem?
ceph fs status
ceph daemon mds.$(hostname -s) perf dump | jq '.mds'

# metadata rate
ceph fs status | grep -A2 'RANK'

# data rate
ceph osd pool stats cephfs-data

A workload complaining about ls and file creation being slow while throughput is fine is entirely an MDS question. One complaining about transfer rates while ls is instant is entirely an OSD question.

Quiz

Knowledge check · 4 questions

Q1. A client reads 40 GB from a file it already has open. How much of that traffic passes through the MDS?

None — the client computes object names and reads directly from the OSDs All of it — the MDS proxies file contents so that it can keep the file size current Roughly half, since the MDS serves the extents it has cached and the OSDs serve the rest Only the first object, which the MDS returns alongside the layout when the file is opened
Q2. A build system creating millions of small files stresses the MDS far more than the data pool.

True False
Q3. Direct an investigation from the symptom.

A CephFS deployment serves both a media team streaming large video files and a CI system building software. The CI team reports that their builds have become very slow; the media team reports no problems at all.
Q4. Why does the MDS hand out inode numbers and layouts rather than proxying data?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Classify every CephFS complaint as metadata or data before investigating; the two paths share almost no diagnostics and the classification eliminates half the possibilities in one question. Size metadata-heavy deployments against MDS capacity and metadata pool latency rather than against data volume, which understates the requirement entirely.

Cross-course references

Kubernetes: control-plane load versus data-plane load is the same separation
Linux: NFS metadata operations versus bulk transfer stress different parts of a server