Quiescing shared-filesystem workloads

Advanced⏱ ~18 minceph

What you'll learn

Recognise why shared filesystems are harder to quiesce
Coordinate across multiple writers
Use per-workload approaches
Decide when to accept inconsistency

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

An RBD image has one writer. A CephFS tree has many, on different hosts, with no shared notion of a moment.

Why it is harder

To quiesce an RBD image you freeze one guest.
To quiesce a CephFS tree you would have to freeze every client writing
to it, simultaneously, from outside.

Difficulty	Consequence
Many clients	no single point to freeze
Clients on different hosts	coordination is a distributed problem
Clients hold capabilities and buffers	flushing requires their cooperation
Clients may be untrusted tenants	you cannot make them cooperate
The snapshot is instant	but the writers are not synchronised

ceph tell mds.0 client ls --format json | python3 -c '
import sys,json
c = json.load(sys.stdin)
print("connected clients:", len(c))
for x in c[:10]:
    md = x.get("client_metadata", {})
    print(" ", md.get("hostname"), md.get("entity_id"))'

What a snapshot gives without coordination

The MDS flushes client capabilities as part of taking the snapshot, so
the snapshot is consistent with respect to what clients had committed —
data still in a client's write buffer is not included.

Content	In the snapshot
Data written and flushed	yes
Data in a client’s page cache, unflushed	no
A file being written at that instant	partially
A multi-file operation half complete	half

So the snapshot is coherent at the filesystem level and says nothing
about application-level operations spanning several files.

Coordinating across writers

Approach	Suits
Pause the workload at its scheduler	build farms, batch pipelines
Application-level checkpoint	anything with its own consistency notion
Quiesce every client via configuration management	small, trusted client sets
Snapshot a subtree that is not being written	archives, completed outputs
Accept filesystem-level consistency	most shared file workloads

# pause the workload, snapshot, resume
systemctl stop build-scheduler
sync
mkdir /mnt/cephfs/builds/.snap/consistent-$(date +%Y%m%d)
systemctl start build-scheduler

# a database on CephFS quiesces itself
psql -c "SELECT pg_backup_start('snap');"
mkdir /mnt/cephfs/pgdata/.snap/consistent
psql -c "SELECT pg_backup_stop();"

Per-workload approaches

Build farm — pause job dispatch briefly; running jobs write to scratch
Home directories — filesystem-level consistency is what users expect
Media pipeline — snapshot completed output directories, not work areas
Container images — the registry has its own consistency
Databases — use the database's mechanism, never the filesystem's alone

# snapshot only the completed area
mkdir /mnt/cephfs/pipeline/output/.snap/daily-$(date +%Y%m%d)

When to accept inconsistency

Accept filesystem-level consistency when:
  the workload has no cross-file transactional requirement
  users understand a backup is "as of roughly then"
  the restore path is per-file rather than whole-tree
  coordinating every writer is not achievable

Quiz

Knowledge check · 4 questions

Q1. What consistency does a CephFS snapshot provide without client coordination?

None at all: the MDS takes the snapshot without involving clients, so anything not already on disk is simply absent from it Consistency with everything clients had committed at that moment — the MDS flushes capabilities — but no application-level consistency across files Full application consistency, because the MDS holds every client's writes in order and cuts the snapshot at a transaction boundary it can identify in the stream Consistency only for files no client had open for writing, with every other file captured mid-write and unusable
Q2. A shared filesystem has no single point at which writes can be frozen, whatever tooling is available.

True False
Q3. Get consistent backups of a build farm on CephFS.

A build farm has 60 nodes writing to a shared CephFS tree. Intermediate build state and finished artefacts share the same directory structure.
Q4. What content is missing from a CephFS snapshot taken without coordination?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Separate work areas from output areas in shared-filesystem workloads — snapshotting only the appended-to output tree gives a free consistency boundary that costs nothing, unlike coordinating dozens of writers. Use the application’s own mechanism for anything transactional.

Cross-course references

Kubernetes: quiescing many pods writing one volume has the same coordination problem
Linux: structuring data so the consistent part is separable beats coordinating writers