Shared CephFS volumes across pods

Why this matters in production

CephFS gives several pods a genuinely shared POSIX filesystem, which is more than most shared storage provides and less than some applications assume.

What is guaranteed

spec:
  accessModes: [ReadWriteMany]
  storageClassName: ceph-fs

The locking that works

import fcntl
with open('/data/shared.lock', 'w') as f:
    fcntl.flock(f, fcntl.LOCK_EX)
    # exclusive across every pod on every node
    fcntl.flock(f, fcntl.LOCK_UN)

# from a shell
flock /data/shared.lock -c 'do-the-thing'

Both flock and fcntl locks are coordinated by the MDS and work across nodes, which is the property that makes CephFS suitable for leader election and coordinated access.

Patterns that fail despite POSIX semantics

# atomic creation, correct
fd = os.open('/data/lock', os.O_CREAT | os.O_EXCL | os.O_WRONLY)

The eviction case

ceph tell mds.0 client ls
ceph osd blocklist ls

A client that loses contact with the MDS is eventually evicted
  → its capabilities are revoked
  → its locks are released
  → another pod can acquire them
  → the evicted pod may still believe it holds the lock

Applications relying on locks for correctness must handle losing them, which means checking a fencing token or re-validating rather than assuming continuity.

Designing for shared volumes

Quiz

Production discipline

Use flock or fcntl for coordination on shared CephFS volumes — both are arbitrated by the MDS and work across nodes — and treat any I/O error as a signal the client may have been evicted and its locks released. Never use file existence as a lock; O_CREAT | O_EXCL is the atomic primitive.

Cross-course references

• Kubernetes: leases and leader election exist because file locks are not sufficient
• Linux: NFS and cluster filesystems face identical lock-recovery semantics