MDS journal damage and recovery

Why this matters in production

Journal damage is rare and the recovery tools are genuinely destructive. Working the sequence in order, and knowing exactly what each step discards, is what separates a recovery from a data-loss event you caused yourself.

Recognising it

ceph fs status cephfs
# rank 0 stuck in replay, or repeatedly crashing there

journalctl -u ceph-mds@a | grep -iE 'journal|corrupt|assert'
ceph tell mds.a damage ls

cephfs-journal-tool --rank=cephfs:0 journal inspect
# Overall journal integrity: DAMAGED

That last command is read-only and is how you confirm rather than assume.

Before anything destructive

# 1. take the filesystem down so nothing is writing
ceph fs fail cephfs

# 2. export the journal — this is the evidence and the fallback
cephfs-journal-tool --rank=cephfs:0 journal export /backup/journal-rank0.bin

# 3. back up the monitor store if not already done

The export is not optional. Every subsequent step is potentially destructive, and the exported journal is the only way to reconsider.

The recovery sequence

Step 1 — recover what the journal still holds.

cephfs-journal-tool --rank=cephfs:0 event recover_dentries summary

This replays recoverable dentry operations into the metadata pool, salvaging metadata the damaged journal still describes. Non-destructive.

Step 2 — reset the journal.

cephfs-journal-tool --rank=cephfs:0 journal reset

Destructive. Discards journal entries not recovered in step 1 — recent creations, deletions, and renames are lost. The filesystem becomes mountable again.

Step 3 — reset the session and inode tables.

cephfs-table-tool all reset session
cephfs-table-tool all reset snap
cephfs-table-tool all reset inode

Clears stale allocation state that would otherwise conflict with the reset journal.

Step 4 — bring the filesystem back.

ceph fs set cephfs joinable true
ceph fs status cephfs

Step 5 — verify and repair.

ceph tell mds.0 scrub start / recursive repair
ceph tell mds.0 scrub status
ceph tell mds.a damage ls

The scrub walks the tree and reconciles metadata against the data pool, finding orphaned inodes and inconsistencies the reset left behind.

If the metadata pool itself is damaged

DATA_POOL=rbd-vms
cephfs-data-scan scan_extents ${DATA_POOL}
cephfs-data-scan scan_inodes ${DATA_POOL}
cephfs-data-scan scan_links
cephfs-data-scan cleanup ${DATA_POOL}

These rebuild metadata by scanning the data pool objects — a last resort that recovers file contents into lost+found with much of the namespace structure gone. It takes a very long time on a large filesystem.

Quiz

Production discipline

Treat every step of this sequence as a decision requiring confirmation rather than a procedure to execute, and make the journal export a mandatory gate before anything destructive. If a support relationship exists, engage it at the export stage — that file is what any serious diagnosis starts from.

Cross-course references

• Kubernetes: etcd disaster recovery follows the same confirm-backup-then-repair sequence
• Linux: filesystem repair tools carry identical warnings for identical reasons