File storage - the shared namespace primitive

Why this matters in production

A database engine owns its filesystem semantics: ext4 or XFS on a raw block device is the right answer. But many production workloads do not own the filesystem - they expect a shared POSIX namespace where multiple clients on different hosts see the same files at the same path at the same time, with the same access controls.

Shared file storage is the primitive for that workload: shared home directories, HPC job scratch space, container image registries (when not backed by object), build farms, and any application whose data model is “a folder full of files”. CephFS is Ceph’s implementation of this primitive on top of RADOS.

What file storage is

A file-storage primitive exposes a hierarchical namespace (folders and files) with POSIX semantics:

• Hierarchical paths. /home/alice/report.txt. Mounted into the kernel VFS, traversed by open(), readdir(), stat().
• POSIX permissions. User, group, world bits; ACLs on top. Same model as a local filesystem; same enforcement.
• File-level operations. open, read, write, seek, fsync, rename, unlink, mkdir, link. A file is a stream of bytes; the filesystem owns directory metadata.
• Atomic-or-not-atomic file writes. Application writes depend on the underlying filesystem’s atomicity guarantees; POSIX defines write behaviour but not durability.

flowchart LR
  app1[App on host A] --> vfs1[VFS - Linux kernel]
  vfs1 --> c1[CephFS fuse or kernel client]
  app2[App on host B] --> vfs2[VFS - Linux kernel]
  vfs2 --> c2[CephFS fuse or kernel client]
  c1 --> mds1[MDS - metadata server]
  c2 --> mds2[MDS - metadata server]
  c1 --> rados[RADOS - file data]
  c2 --> rados
  mds1 --> rados
  rados --> osds[OSDs]

When file storage is the right answer

• The application expects a filesystem. Many software packages assume a path-based interface: build systems, image processing pipelines, log shippers writing structured files, scientific computing.
• Multiple clients read and write the same namespace. This is the killer feature: a POSIX-compliant shared filesystem mounts on every client; every client sees the same file at the same time; locks (POSIX byte-range locks or fcntl leases) coordinate concurrent writers.
• A workload combines small random reads with file-create and file-delete. A code-search tool reading millions of small files is the canonical example: object storage is too expensive per request.

When file storage is the wrong answer

• Single-writer tight latency workloads. Use block storage.
• Many-TiB immutable archives. Object storage is cheaper.
• Write-once-read-many workloads where per-object API matters more than POSIX semantics. Object storage.

How CephFS implements the file primitive

CephFS is a POSIX-compliant filesystem built on RADOS. Two main components:

• MDS (ceph-mds). The metadata server. Stores and serves the CephFS inode table, directory structure, and locking state. Without an MDS, no metadata; data alone is not enough to give a POSIX namespace.
• RADOS data pool. The actual file contents are stored as Ceph objects, one per file (or one per object the file is striped into, depending on striping configuration).

Multiple MDS daemons can run as active/standby or as a distributed metadata cluster for very large namespaces.

flowchart TB
  client[CephFS client] --> mds_meta[Query MDS for directory/inode]
  client --> rados_data[Read/write file data from RADOS]
  mds_meta --> mds_daemon[MDS daemon]
  mds_daemon --> mds_pool[MDS metadata pool]
  rados_data --> data_pool[Data pool]

CephFS mounting

A client mounts CephFS via the kernel ceph module (mount -t ceph ...) or via FUSE (ceph-fuse). Both modes speak the same MDS protocol. The kernel mount is faster; FUSE is easier to troubleshoot and works in restricted namespaces.

# Kernel mount (preferred for performance)
mount -t ceph ceph-mon-1:6789,ceph-mon-2:6789,ceph-mon-3:6789:/ /mnt/cephfs \
 -o name=client.cifs,secretfile=/etc/ceph/client.cifs.secret

# FUSE mount (preferred for diagnostics)
ceph-fuse -m ceph-mon-1:6789 /mnt/cephfs

File storage vs. block in the Ceph context

A common production confusion is to compare RBD to CephFS and conclude they are interchangeable. They are not:

Both are RADOS underneath. Both are Ceph services on the same cluster. The choice between them is a workload choice, not a performance choice.

Quiz

Production discipline

CephFS is the right primitive when multiple clients need a shared POSIX namespace. The MDS is the operational centre of gravity: if the MDS is unhealthy, the filesystem is unhealthy. Plan MDS capacity, MDS redundancy, and MDS failover behaviour the same way you would plan a database primary - CephFS is as availability-critical as any consumer of the service. Where single-writer block I/O is the right answer (databases, VM disks), use RBD; where shared writes are required, use CephFS; never mount the same RBD image on multiple hosts.

Cross-course references

• Linux: Part XIV (Filesystems), Part XXXVIII (Linux Performance Fundamentals), Part LIX (Shared Storage and Clusters).
• Proxmox: Part VII (Shared Storage) for NFS/iSCSI, Part VIII (Ceph) for CephFS specifics.
• Kubernetes: Part LII (CSI), Part LIII (Storage Failure Modes).