Lab: Build a fstab entry from scratch and recover from a broken one

This lab walks through the full lifecycle of an fstab entry: design, validation, and recovery. By the end you will have written a working entry, broken it deliberately, and recovered.

Objective

By the end of this lab, you can:

Design an fstab entry that survives hardware changes.
Validate it with mount -a before any reboot.
Recover a host that fails to boot because of a bad fstab.
Apply preventive options (nofail, _netdev) correctly.

Tasks

Task 1: Design an fstab entry

Use a loop-backed image. It behaves like a real block device for everything this lab needs, and its blast radius is one file you delete at the end.

# Create the backing file FIRST - losetup does not create it
truncate -s 1G /tmp/data.img

# --nooverlap re-uses an existing device for this file rather than
# attaching a second one to the same image
LOOP=$(sudo losetup --find --nooverlap --show /tmp/data.img)
echo "attached: $LOOP"

sudo mkfs.ext4 -L data "$LOOP"
UUID=$(sudo blkid -s UUID -o value "$LOOP")
echo "UUID=$UUID /data ext4 defaults,nofail 0 2"

If you use a real partition instead, prove it is empty first

mkfs.ext4 writes a new filesystem over whatever is on the target. It does not ask, and lsblk is not the check: it lists devices and mount points, and an unmounted partition full of data looks exactly like a spare one.

Two commands establish that a partition is genuinely free. Both must print nothing:

lsblk -o NAME,SIZE,FSTYPE,MOUNTPOINT,LABEL     # orient yourself
DEV=/dev/sdXN                                   # YOUR spare partition

sudo blkid "$DEV"          # STOP if this prints a TYPE or UUID - there is a filesystem here
findmnt --source "$DEV"    # STOP if this prints anything - it is mounted somewhere
sudo wipefs "$DEV"         # STOP if this lists signatures (read-only without -a)

sudo mkfs.ext4 -L data "$DEV"

A device name is not a stable identity either. /dev/sdb is assigned in discovery order and can refer to a different disk after a reboot or a controller change — which is the whole reason this lab teaches UUIDs. Confirm the size and model with lsblk -o NAME,SIZE,MODEL,SERIAL before you type a device path into mkfs.

Write down the entry you would put in /etc/fstab, with rationale for:

The choice of UUID vs LABEL.
The mount options (especially nofail, noatime, errors=…).
The pass number (0 vs 2).
The dump number (always 0 in production).

Task 2: Validate the entry

# backup current fstab
cp /etc/fstab /etc/fstab.backup
# add the new entry
echo "UUID=... /data ext4 defaults,nofail 0 2" >> /etc/fstab
# 1. parse and sanity-check EVERY entry, including mounted ones
findmnt --verify --verbose; echo "exit=$?"
# 2. regenerate systemd's .mount units from the edited file
systemctl daemon-reload
# 3. mount whatever is not mounted yet
mount -a
# verify
findmnt /data

Validate in that order, and do not stop at step 3. mount -a succeeding is not proof the host will boot:

It skips any entry whose source and target are already mounted, so a wrong UUID on a live mount is invisible to it.
It skips noauto entries by design.
nofail tells mount not to report an error when the device is missing, so a broken entry passes silently.

findmnt --verify checks every entry including the mounted ones, mounts nothing, and still reports [E] on an entry carrying nofail. Its exit status is what you gate on: non-zero means at least one entry would fail at boot.

Task 3: Break the entry deliberately

Pick a safe mount point like /data. Edit the entry to use an incorrect UUID, and drop nofail for this step so the failure is allowed to surface:

# change the UUID to a wrong value
sed -i 's/^UUID=correct-uuid/UUID=00000000-0000-0000-0000-000000000000/' /etc/fstab
# edit the same line by hand and change defaults,nofail to defaults
vi /etc/fstab
grep ' /data ' /etc/fstab
# UUID=00000000-0000-0000-0000-000000000000 /data ext4 defaults 0 2

Unmount first. This is the whole point of the exercise: while /data is still mounted, mount -a sees the target as done and skips the entry entirely, so the broken UUID produces no error at all.

umount /data
mount -a; echo "exit=$?"
# mount: /data: can't find UUID=00000000-0000-0000-0000-000000000000.
# exit=1

The message names the unresolvable identifier. It is not wrong fs type - that message means the device was found but its filesystem could not be recognised, which is a different fault with a different fix.

Now run the check that would have caught this without unmounting anything:

findmnt --verify --verbose; echo "exit=$?"
# 0 parse errors, 1 error, 0 warnings
# /data
#    [ ] target exists
#    [E] unreachable on boot required source: UUID=00000000-0000-0000-0000-000000000000
# exit=1

Put nofail back on the entry and run findmnt --verify again: the [E] line and the non-zero exit stay. That is why it is the gate and mount -a is not.

The mount fails at boot, so the system is now in a state where fstab would prevent boot. With nofail restored on a non-root mount, the host continues booting without /data - degraded, not down. For the recovery drill, do the same to the root mount on a clone VM, where nofail is not an option and the host drops to emergency mode.

Task 4: Recover via emergency mode

Reboot the clone VM. It will hang at the broken fstab entry.

In a hypervisor console, interrupt the boot and edit the GRUB entry. Append systemd.unit=emergency.target to the linux line. The system drops to an emergency shell.

From the emergency shell:

mount -o remount,rw /
vi /etc/fstab
# fix the entry
reboot

The host should now boot normally with the fixed fstab.

Task 5: Document

Document:

The original fstab entry and its rationale.
The mount -a output.
The recovery walkthrough with timestamps.
The prevention measures: UUID usage, nofail for non-critical, mount -a testing, and a documented runbook for the recovery procedure.

Validation

A working fstab entry exists with UUID and rationale for each field.
findmnt --verify reports no errors and exits 0.
systemctl daemon-reload has been run since the last edit.
mount -a succeeds.
The broken-entry run produced can't find UUID=... from mount -a after umount /data, and [E] unreachable on boot from findmnt --verify.
The recovery procedure was executed successfully.
Documentation includes the runbook entry for future operators.

Cleanup

Restore the original fstab if you modified it:

cp /etc/fstab.backup /etc/fstab
mount -a

Detach the loop device this lab created — and only that one:

# Find the device backed by our image, rather than detaching everything
sudo losetup --associated /tmp/data.img
sudo losetup --detach $(sudo losetup --associated /tmp/data.img -O NAME --noheadings)
rm -f /tmp/data.img

# Confirm nothing of ours is left
sudo losetup --associated /tmp/data.img    # no output

losetup -D detaches every loop device on the host. On a machine that also runs snap packages (each mounted from a loop device), a live ISO, or any service using a loop-backed image, that pulls the filesystem out from under them. Detach by name.

What you learned

An fstab entry has six columns, and the choice of device identifier (UUID, LABEL) determines whether the host boots after hardware changes.
mount -a is the last step of the pre-reboot check, not the whole of it. It skips entries that are already mounted, skips noauto, and stays silent on a missing device when the entry carries nofail. Gate on findmnt --verify first, then systemctl daemon-reload, then mount -a.
The recovery path is: get to emergency mode, remount read-write, fix fstab, reboot. This works for the vast majority of fstab issues.
For non-critical mounts, nofail prevents a single failure from blocking boot.

Objectives

Prerequisites