Recovering from a broken fstab without a live USB · Linux

A bad fstab entry - wrong UUID, missing filesystem, typo in the mount point - prevents the host from booting. The recovery path is the same: get to an emergency shell, mount the root filesystem read-write, fix fstab, reboot.

Diagnosing the failure

Read-only / Safeverify fstab— findmnt --verify parses every entry in /etc/fstab and checks it for usability: does the target exist, does the UUID or LABEL resolve to a device, is the filesystem type known, are the options parsable. It checks entries that are already mounted, which is exactly the case mount -a cannot test, and it still reports [E] on an entry carrying nofail. It mounts nothing, so it is safe on a live host. Check the exit status: non-zero means at least one entry failed.

$ findmnt --verify --verbose

0 parse errors, 1 error, 0 warnings
/data
[ ] target exists
[ ] userspace options: nofail
[E] unreachable on boot required source: UUID=00000000-0000-0000-0000-000000000000

Configuration changeregenerate mount units— Re-runs systemd-fstab-generator so systemd's .mount units match the file you just edited. Without this, systemctl and systemd's boot-time view still reflect the previous fstab. Follow it with systemctl list-units --type=mount to see the regenerated units.

$ sudo systemctl daemon-reload

Configuration changemount the rest— Last step, not the only step. It proves that entries which are not yet mounted can be mounted now. Combined with findmnt --verify it is a reasonable pre-reboot gate. On its own it is not.

$ sudo mount -a

Recovering via systemd emergency mode

Data-loss riskreboot to firmware— Reboot into UEFI/BIOS firmware settings. From the firmware menu, select boot options, then add a temporary boot entry that loads the kernel with systemd.unit=emergency.target. This is the cleanest way to get to emergency mode for fixing fstab.

$ systemctl reboot --firmware-setup

Data-loss riskGRUB emergency— Edit the GRUB entry for the default boot. Append systemd.unit=emergency.target to the linux line. systemd will drop to a root shell with the root filesystem mounted read-only. Remount read-write, fix fstab, reboot.

$ systemd.unit=emergency.target

Once in emergency mode, read the kernel log before you remount anything:

Read-only / Safewhy is / read-only?— Two very different states look identical at this prompt. Emergency mode mounts / read-only on purpose, and remounting it read-write is the normal next step. A filesystem the kernel demoted mid-life with errors=remount-ro is the kernel protecting your data after a metadata or I/O error. No hits means the first case: proceed. Any hit means the second: do not remount, and treat it as a storage incident.

$ journalctl -k -b --no-pager | grep -iE 'EXT4-fs error|XFS.*(Corruption|Internal error)|Buffer I/O error|blk_update_request|medium error|Remounting filesystem read-only'

If that grep is clean, the read-only root is just emergency.target and the remount is safe:

Configuration changeremount rw— The root filesystem is mounted read-only in emergency mode. Remount it read-write to edit files: mount -o remount,rw /. Only run this once the journalctl -k grep above has come back clean.

$ mount -o remount,rw /

A blind remount,rw can be the thing that loses the data

mount -o remount,rw / is the correct step in emergency mode and the wrong one on a host whose root was demoted by errors=remount-ro. The command is identical; only the kernel log tells you which situation you are in.

The wrong version reads: operator sees a read-only root, remounts read-write, edits /etc/fstab, the host runs for twenty minutes and the filesystem fails again — this time with the metadata damaged past fsck, and with the original dmesg evidence rolled out of the ring buffer.

If the grep does hit, stop treating this as an fstab problem. Confirm with findmnt -o TARGET,SOURCE,FSTYPE,OPTIONS /, copy dmesg -T output off the host, identify the device with lsblk -o NAME,SIZE,SERIAL,MOUNTPOINT, and check the media with smartctl -a (or nvme smart-log). Repair happens unmounted, from rescue media, after the volume has been imaged — never by remounting read-write to buy time. The full triage is in linux-boot-failure-diagnosis and in Mode 3b of linux-runbook-recover-failed-boot.

Configuration changefix fstab— Fix the offending fstab entry. Common fixes: replace device path with UUID; correct typos; comment out non-essential mounts with nofail so the boot continues; ensure mount points exist. Save and exit.

$ vi /etc/fstab

Data-loss riskreboot— Reboot normally. The host should come up with the corrected fstab. If it does not, repeat the recovery cycle - emergency shell is always available.

$ reboot

Document the fstab error and the fix in the post-incident review
Add a nofail option to non-critical mounts in fstab to prevent future boot hangs
Set up a runbook with the emergency shell procedure
Test the recovery procedure on a clone host before relying on it in production

Recovery via the initramfs shell

If the root filesystem itself cannot be mounted, you must stop the boot before the pivot to the real root. That is the initramfs shell, and it is reached with a distribution-specific kernel parameter.

The real initramfs shell - root will not mount

Data-loss riskdracut initramfs shell— RHEL, Fedora, CentOS Stream, SUSE - anything using dracut. Append rd.break to the linux line in GRUB. The initramfs stops just before switch-root and gives you a shell. Variants let you stop earlier: rd.break=pre-mount stops before the root is mounted at all. See man 7 dracut.cmdline.

$ rd.break

Data-loss riskinitramfs-tools shell— Debian and Ubuntu - anything using initramfs-tools. Append break=mount to the linux line to stop just before the root filesystem is mounted. A bare break= stops at the earliest hook. The prompt is (initramfs).

$ break=mount

From that shell the real root is not yet mounted, so there is no /etc/fstab to fix yet. The job is to find the root device and bring it online:

blkid - list every block device with its UUID, LABEL and filesystem type. Compare against the root= parameter the bootloader passed.
lvm vgscan && lvm vgchange -ay - activate volume groups if the root is on LVM. A root that "cannot be mounted" is very often just an inactive VG.
mdadm --assemble --scan - assemble software RAID arrays if the root is on md.
cryptsetup luksOpen /dev/sdaN root - unlock the root if it is encrypted.
mount -o ro /dev/<root-device> /sysroot - mount the root read-only first, so a damaged filesystem is not written to before you have looked at it.
Inspect /sysroot/etc/fstab. If it is the problem, remount read-write with mount -o remount,rw /sysroot and edit it.
exit - the initramfs continues the boot from where it stopped.

init=/bin/bash - root mounts, userspace is broken

Use this when emergency mode itself will not start - a corrupted journal, a broken systemd unit, a mangled /etc/passwd. The root filesystem must still be mountable.

The kernel starts bash as PID 1 with nothing else set up. No /proc, no /sys, no services, and the root mounted read-only. mount -o remount,rw / fails with a confusing error until /proc is mounted, because mount reads /proc/self/mountinfo. Mount the pseudo-filesystems first:

Configuration changeprepare the PID 1 shell— Under init=/bin/bash nothing has been mounted for you. Mount /proc and /sys before anything else, then remount the root read-write. Skipping this is why 'mount -o remount,rw /' appears to be broken in this mode.

# mount -t proc proc /proc && mount -t sysfs sys /sys && mount -o remount,rw /

Data-loss riskhand back to systemd— When you are done, flush the page cache with sync and drop the root back to read-only before handing control to systemd, otherwise systemd re-mounts a filesystem it believes is clean and you can lose the edits or leave the filesystem inconsistent. exec /sbin/init continues the boot; reboot -f is the safer alternative if anything felt unstable. Never use a plain 'reboot' here - there is no systemd to receive it.

# sync && mount -o remount,ro / && exec /sbin/init

Common fstab mistakes

Mistake	Symptom	Fix
`/dev/sda1` changed to `/dev/sdb1` after hardware change	Host does not boot	Use UUID instead
Mount point does not exist	mount fails at boot	Create the directory or fix the path
Filesystem type wrong	mount fails	Use `auto` to let the kernel detect, or set the correct type
Missing `nofail` on a network mount	Host hangs at boot	Add `nofail` so boot continues, or fix the network
Typo in UUID	mount fails	`blkid` to find the right UUID
Missing `0 2` at the end	Nothing breaks: fields 5 and 6 are optional and default to 0. But dump=0 fsck=0 means the filesystem is never checked at boot, so corruption accumulates unnoticed	Write all six columns. Use fsck pass 1 for `/` and 2 for other local filesystems; 0 for network and bind mounts