Lab: Filesystem check and repair with fsck and xfs_repair

What this lab is

You will work through the safe sequence for three real situations:

1. An ext4 filesystem that was not cleanly unmounted, with possible metadata corruption. The right answer is e2fsck -n first, then e2fsck -p (preen), then interactive -y only if you understand what is being repaired.
2. An XFS filesystem that mounts read-only or refuses to mount because the log is dirty. The right answer is xfs_repair -n to inspect, then mount -o ro,norecovery to read, then xfs_repair -L only if a fresh backup does not exist.
3. A device that is failing at the hardware layer (reallocated sectors, I/O errors). The right answer is not to repair at all until the device is replaced; the audit caught a reader who lost data because they accepted every e2fsck suggestion on a disk with unreadable blocks.

The single rule that runs through all three: image before you write. A repair is a write to the same device the metadata was on. If the hardware is faulty, the repair can destroy recoverable data. Take a snapshot first.

Setup

A nested VM with two scratch block devices:

• /dev/vdb — 1 GiB, no filesystem, used for the ext4 sequence.
• /dev/vdc — 1 GiB, no filesystem, used for the XFS sequence.

Both are fresh; the lab writes a filesystem, deliberately breaks a piece of metadata, and asks you to detect and repair it.

# Confirm the scratch devices.
lsblk -o NAME,SIZE,WWN,SERIAL,MOUNTPOINTS,FSTYPE /dev/vdb /dev/vdc

# Both should be empty and unclaimed.
wipefs /dev/vdb
wipefs /dev/vdc

If you are following this lab on a real host that already has a filesystem on these devices, stop and pick a different pair. The lab destroys the filesystem on each device.

Step 1 — ext4: write, break, repair [UNVERIFIED]

DEV=/dev/vdb
mkfs.ext4 -L scratch "$DEV"
mkdir -p /mnt/scratch
mount "$DEV" /mnt/scratch
# Write enough data that e2fsck has something to look at.
dd if=/dev/urandom of=/mnt/scratch/file bs=1M count=100 status=none
sync
umount /mnt/scratch

Confirm the filesystem is clean, then deliberately break it:

umount "$DEV"
# Corrupt the superblock by overwriting its primary copy with zeros.
# `seek` and `count` here are in 512-byte sectors; the superblock sits
# at offset 1024 bytes (2 sectors) from the start of an ext4 device.
dd if=/dev/zero of="$DEV" bs=512 count=8 seek=2 conv=notrunc status=none

Now inspect:

e2fsck -n "$DEV"

-n answers every question “no” without writing. It will report “Superblock has an invalid journal” or similar. The point of this run is to read, not to repair. Record the output.

# Use a backup superblock to mount -o ro and read the data back.
mkfs.ext4 -n "$DEV"   # note the backup superblock offsets e2fsck prints
e2fsck -b 32768 "$DEV"

If -b 32768 succeeds without -n, e2fsck is repairing with the backup superblock. This is destructive. Re-image first.

Step 2 — XFS: write, dirty the log, repair [UNVERIFIED]

DEV=/dev/vdc
mkfs.xfs -L scratch "$DEV"
mkdir -p /mnt/scratch
mount "$DEV" /mnt/scratch
dd if=/dev/urandom of=/mnt/scratch/file bs=1M count=100 status=none
sync
# Dirty the log by pulling the device without unmounting.
# (Do NOT do this on a real production device. This is the lab.)
echo "skipping the simulated dirty-log scenario in B-nested mode"
umount /mnt/scratch

The clean procedure for a real XFS dirty-log scenario:

# Step 2a: never `xfs_repair` a mounted filesystem.
mount "$DEV" /mnt/scratch || echo "mount failed - dirty log"

# Step 2b: read-only mount with norecovery to read what's there.
mount -o ro,norecovery "$DEV" /mnt/scratch
ls /mnt/scratch
umount /mnt/scratch

# Step 2c: xfs_repair -n for a read-only inspection.
xfs_repair -n "$DEV"

# Step 2d: only if 2a-2c show no application-consistent backup and the
# data is acceptable to lose, run xfs_repair -L. The -L flag *zeroes*
# the log; it does not replay it. Anything written since the last
# log checkpoint is lost.
xfs_repair -L "$DEV"
mount "$DEV" /mnt/scratch

xfs_repair -L is the loud button. Its docstring is one sentence long and says exactly this. If you find yourself reaching for it on production data, take a backup and restore instead.

Step 3 — hardware fault: do not repair [UNVERIFIED]

A disk that returns I/O errors is not a filesystem problem. It is a hardware problem. The procedure is:

# Step 3a: capture the kernel's view of the device.
dmesg | tail -50
journalctl -k --since "5 min ago"

# Step 3b: get the SMART data and read errors.
smartctl -a /dev/sdX
smartctl -l selftest /dev/sdX

# Step 3c: stop repairing. Replace the disk. Re-image from backup.

The trap to avoid: a filesystem with hardware-level corruption can mount and look fine until the corrupted region is read. Repair tools that fix what the disk returns may paper over the corruption without flagging it, leaving the reader to discover the loss weeks later.

Acceptance criteria

• e2fsck -n output captured for the corrupt ext4 sequence.
• A backup-superblock repair run on an LVM snapshot, not on the live device.
• xfs_repair -n output captured for the dirty-log sequence.
• A repair-vs-replace decision recorded for the hardware-fault case, with smartctl -a output.
• A short note on which step the maintainer needs to run end-to-end before removing the [UNVERIFIED] markers.