The MDS journal and why recovery works

Expert⏱ ~18 mincephcephfs-journal-tool

What you'll learn

Explain the MDS write-ahead journalling model
Describe journal replay on MDS restart
Assess the effect of journal size on recovery time
Inspect and, in emergencies, repair a journal

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

The journal is why an MDS can die without losing metadata, and its size is the main determinant of how long recovery takes. Both facts matter during an incident, and the second is a tunable most people never touch.

The model

1. client → MDS:  create /data/report.txt
2. MDS:           append the operation to the journal (a RADOS object)
3. MDS:           wait for the journal write to commit
4. MDS → client:  success
5. MDS:           apply the change to its in-memory cache
6. MDS:           later, flush cached metadata to the metadata pool

The acknowledgement comes after step 3, so the operation is durable in the journal before the client is told it succeeded. Steps 5 and 6 can happen much later.

The journal lives in the metadata pool as a stream of objects:

rados -p cephfs-meta ls | grep -E '^2[0-9a-f]*\.' | head

Replay on restart

When an MDS starts or takes over a rank:

1. read the journal from its last known checkpoint
2. replay every operation into memory
3. rebuild the cache to a consistent state
4. resume serving

ceph fs status cephfs
# RANK  STATE
#  0    replay      mds.b
#  ...later...
#  0    active      mds.b

Replay duration is proportional to journal length. A large journal means a long replay, during which the filesystem is unavailable for that rank.

Journal size

ceph config get mds mds_log_max_segments
ceph config get mds mds_log_max_events

Larger journal	Smaller journal
Fewer metadata pool flushes	more frequent flushes
Better write batching	less batching
Longer replay on failover	faster failover

The trade is throughput against recovery time. A deployment where MDS failover latency matters should keep the journal smaller; one optimising metadata write throughput can afford it larger.

Inspecting

cephfs-journal-tool --rank=cephfs:0 journal inspect
# Overall journal integrity: OK

cephfs-journal-tool --rank=cephfs:0 header get
cephfs-journal-tool --rank=cephfs:0 event get summary

These are read-only and safe to run against a healthy filesystem.

Emergency repair

# only with the filesystem down and after a backup
cephfs-journal-tool --rank=cephfs:0 journal export /backup/journal.bin
cephfs-journal-tool --rank=cephfs:0 event recover_dentries summary
cephfs-journal-tool --rank=cephfs:0 journal reset

journal reset discards unapplied metadata operations. It is a last resort that loses recent metadata changes, and it must be preceded by an export.

Quiz

Knowledge check · 4 questions

Q1. When does an MDS acknowledge a metadata operation to the client?

After the journal write commits to RADOS, before the change is flushed to the metadata pool After the change has been applied to the metadata pool objects themselves Immediately, with the journal write following asynchronously behind the reply After every replica in the metadata pool has acknowledged the flush of the change
Q2. A larger MDS journal improves metadata write throughput but lengthens failover recovery.

True False
Q3. Reduce MDS failover time for a latency-sensitive deployment.

A CephFS deployment serving an interactive application takes about 90 seconds to complete MDS failover, during which the filesystem is unavailable. The application's users notice every occurrence. Metadata write throughput is well within capacity.
Q4. Why does the MDS journal live in RADOS rather than on the MDS host's local disk?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Measure actual MDS failover time and tune journal size against the availability requirement, since the default trades recovery time for throughput many deployments do not need. Keep cephfs-journal-tool write operations strictly behind a filesystem-down procedure with a mandatory export — they are recovery tools with no safety interlock.

Cross-course references

Kubernetes: etcd WAL replay on restart follows the same model with the same size trade-off
Linux: filesystem journal size versus recovery time is the identical consideration