Skip to main content
RunBook Academy

← All break/fix scenarios in Linux

advancedBoot~40 min

Break/Fix: the host boots to emergency mode but `mount -a` works fine

Reported symptoms

  • ●The host boots to `Give root password for maintenance` after a routine fstab edit
  • ●On Debian and Ubuntu the console instead reads `Cannot open access to console, the root account is locked`
  • ●At the emergency prompt `mount -a` succeeds with no error and `df` shows every filesystem mounted
  • ●`systemctl default` from that prompt brings the system fully up and it runs normally for weeks
  • ●The next reboot lands in emergency mode again
  • ●Nothing in the boot output mentions fstab, and `grep -i fstab` over the journal returns nothing

Evidence

  • · `sudo journalctl -xb -p err` shows `Timed out waiting for device /dev/disk/by-uuid/...`
  • · `sudo journalctl -xb` shows `Dependency failed for /srv/backup` then `Dependency failed for Local File Systems`
  • · `systemctl list-units --type=mount --state=failed` lists `srv-backup.mount`
  • · `systemctl status local-fs.target` reports the target as failed
  • · `sudo blkid` at the emergency prompt lists the UUID from fstab, present and correct
  • · `sudo findmnt --verify --verbose` reports the entry as unreachable at boot but syntactically valid
  • · `sudo dmesg -T | grep -E "sdd|scsi 8"` shows the enclosure attaching about twelve seconds after the device job timed out
  • · `systemctl show -p DefaultTimeoutStartSec` reports 90s
Diagnosis and resolutionclick to reveal

Root cause

The fstab entry is correct and the device is real; it just is not there yet when systemd looks for it. systemd-fstab-generator turns every fstab line into a `.mount` unit, and a mount without `nofail` becomes a hard requirement of `local-fs.target`. The generated unit waits on the corresponding `.device` unit, which udev only publishes once the device appears. The external enclosure holding this filesystem enumerates about 110 seconds into boot, past the 90-second device timeout, so the `.device` unit times out, the `.mount` unit fails, `local-fs.target` fails, and systemd falls back to `emergency.target` — by design, because it cannot know whether the missing filesystem was important. `mount -a` at the emergency prompt succeeds because by the time an operator is typing, the device has been present for minutes. That is why the entry looks innocent and why `mount -a` is not a test of whether a host will boot.

Remediation

Decide whether the mount is required for the system to be considered booted. If it is not, add `nofail` so a missing device degrades the boot instead of stopping it. If the device is merely slow, raise its wait with `x-systemd.device-timeout=` to something above its observed enumeration time. In this case use both — `nofail,x-systemd.device-timeout=180` — so a slow enclosure is waited for and a genuinely absent one does not take the host down. Run `sudo systemctl daemon-reload` so the generator re-runs, then `sudo findmnt --verify --verbose` for syntax, and confirm with a real reboot: the generator only runs at boot and on daemon-reload, so an edited fstab that has never been reloaded is untested.

Verification

After `sudo systemctl daemon-reload`, `systemctl cat srv-backup.mount` must show the new options in the regenerated unit. Reboot the host — this is the only verification that can actually fail, and it is the one that was skipped. After the reboot `systemctl is-system-running` must return `running` rather than `degraded` or `maintenance`, `systemctl status local-fs.target` must be active, and `findmnt /srv/backup` must show the filesystem mounted. Then prove the `nofail` behaviour deliberately: detach the enclosure, reboot, and confirm the host reaches multi-user with the mount absent instead of dropping to emergency.

Prevention

Treat every fstab edit as a change that is untested until the host has rebooted. Add `nofail` to every mount that is not required for the system to function, and reserve a bare entry for the filesystems whose absence genuinely should stop the boot. Run `sudo findmnt --verify --verbose` before rebooting to catch the syntax class of error, and remember it cannot catch the timing class. Set a root password or an alternative recovery path before you need one — a host with a locked root account and a failed `local-fs.target` needs console access and a kernel-command-line edit, not a password. Where a device is genuinely slow, fix the enumeration rather than only raising the timeout.

Reported symptoms

  • A backup disk in an external enclosure was added to /etc/fstab on Tuesday. The change was reviewed, mount -a was run, the filesystem mounted, and the ticket was closed.
  • The host was rebooted on Friday for a kernel update and came up in emergency mode.
  • At the emergency prompt mount -a runs clean and df shows every filesystem including /srv/backup. systemctl default brings the host fully up.
  • The operator concluded fstab was fine — after all, it mounted — and rebooted to confirm. It went to emergency mode again.
  • Searching the journal for fstab returns nothing at all.

Evidence provided

$ sudo journalctl -xb -p err --no-pager
Aug 11 09:02:41 host01 systemd[1]: dev-disk-by\x2duuid-3f9a...device: Job dev-disk-by\x2duuid-3f9a...device/start timed out.
Aug 11 09:02:41 host01 systemd[1]: Timed out waiting for device /dev/disk/by-uuid/3f9a1c7e-2b40-4d1a-9f3e-8c7d6e5b4a21.
Aug 11 09:02:41 host01 systemd[1]: Dependency failed for /srv/backup.
Aug 11 09:02:41 host01 systemd[1]: Dependency failed for Local File Systems.
Aug 11 09:02:41 host01 systemd[1]: emergency.target: Trigger limit hit, refusing further activation.

$ systemctl list-units --type=mount --state=failed
  UNIT             LOAD   ACTIVE SUB    DESCRIPTION
* srv-backup.mount loaded failed failed /srv/backup

$ systemctl status local-fs.target
* local-fs.target - Local File Systems
     Loaded: loaded (/usr/lib/systemd/system/local-fs.target; static)
     Active: failed

$ sudo blkid | grep 3f9a1c7e
/dev/sdd1: UUID="3f9a1c7e-2b40-4d1a-9f3e-8c7d6e5b4a21" TYPE="xfs" PARTUUID="a1b2c3d4-01"

$ grep backup /etc/fstab
UUID=3f9a1c7e-2b40-4d1a-9f3e-8c7d6e5b4a21  /srv/backup  xfs  defaults  0  2

$ sudo dmesg -T | grep -E 'sdd|scsi 8' | head -3
[Tue Aug 11 09:02:51 2026] scsi 8:0:0:0: Direct-Access     ENCL     ARRAY            0210 PQ: 0 ANSI: 6
[Tue Aug 11 09:02:53 2026] sd 8:0:0:0: [sdd] 15628053168 512-byte logical blocks
[Tue Aug 11 09:02:53 2026] sd 8:0:0:0: [sdd] Attached SCSI disk

$ systemctl show -p DefaultTimeoutStartSec
DefaultTimeoutStartSec=1min 30s

Work the evidence before reading on

The journal never says “fstab” because systemd does not think in fstab — it thinks in units. Work out these three before continuing:

  1. Compare the timestamp on Timed out waiting for device with the timestamp on Attached SCSI disk. Which came first, and by how long?
  2. blkid finds the UUID. The device exists. So what exactly timed out — the mount, or something else?
  3. The fstab line says defaults. Write down what defaults does not include before you read the next section.

Root cause

1. fstab is not read at mount time. It is compiled at boot time

systemd-fstab-generator runs early in boot (and again on every daemon-reload) and turns each fstab line into a generated .mount unit under /run/systemd/generator/. You can read the result:

systemctl cat srv-backup.mount

That unit gains a Requires= and After= on the .device unit for the backing device — here dev-disk-by\x2duuid-3f9a1c7e...device. udev publishes that .device unit only when the device node and its by-uuid symlink actually exist.

2. defaults does not include nofail

defaults in fstab means rw,suid,dev,exec,auto,nouser,async. It says nothing about failure behaviour. A mount unit generated from a line without nofail is pulled in by local-fs.target with Requires=, which means:

  • the target waits for it, and
  • if it fails, the target fails.

With nofail, the generator uses Wants= instead and adds nofail semantics: the mount is attempted, and a failure is logged and tolerated.

3. local-fs.target failing sends the system to emergency

This is deliberate and it is the right default. systemd cannot know whether the filesystem that did not appear held /var, a database, or a backup archive nobody will miss until Sunday. Rather than start services that may write to the mount point — filling the root filesystem with data that was supposed to land on the array — it stops and asks a human.

4. The enclosure is 20 seconds too slow

Attached SCSI disk lands at 09:02:53. The device job timed out at 09:02:41, twelve seconds earlier, having waited the default 90 seconds from the point systemd started the job. The enclosure spins up, negotiates, and enumerates on its own schedule, and that schedule is longer than systemd’s patience.

5. Why the recovery prompt was itself a problem

On Debian and Ubuntu the root account is locked by default. sulogin will not open a shell for a locked account, so instead of a maintenance prompt the console prints:

Cannot open access to console, the root account is locked.
See sulogin(8) man page for more details.
Press Enter to continue.

At that point the only way in is the boot loader: interrupt GRUB, append a kernel argument, and boot to a shell.

Resolution

  1. Get a shell. If the emergency prompt accepts a root password, use it. If root is locked, interrupt GRUB, press e on the default entry, and append to the linux line:
  2. `` systemd.unit=rescue.target ``
  3. If rescue.target also depends on the failed target, append init=/bin/bash instead and remount root writable once you have a prompt:
  4. `` mount -o remount,rw / ``
  5. Identify the failed unit rather than reading fstab. systemctl list-units --type=mount --state=failed names it; systemctl status on that unit gives the reason. Reading fstab tells you what you wrote, not what systemd did with it
  6. Amend the option field. Replace defaults with an option set that says what you actually mean:
  7. `` UUID=3f9a1c7e-2b40-4d1a-9f3e-8c7d6e5b4a21 /srv/backup xfs defaults,nofail,x-systemd.device-timeout=180 0 2 ``
  8. Regenerate the units. sudo systemctl daemon-reload re-runs the generator. Confirm with systemctl cat srv-backup.mount that the options landed — if they did not, the edit has not taken effect and the reboot will fail the same way
  9. Reboot. Not mount -a, not systemctl default. Reboot
  10. Consider fixing the enumeration too. A 110-second enclosure is worth a firmware check or a staggered-spinup setting. Raising a timeout accommodates the problem; it does not solve it

Verification

  1. The generated unit carries the new options. systemctl cat srv-backup.mount shows nofail and the device timeout after daemon-reload
  2. The host reaches multi-user unattended. Reboot and confirm systemctl is-system-running returns running. degraded means something else failed; maintenance means you are back in emergency
  3. The filesystem is actually mounted. findmnt /srv/backup returns the mount with the expected source and options. A boot that succeeds because the mount was quietly skipped is not a fix
  4. **local-fs.target is active.** systemctl status local-fs.target shows active, and systemctl list-units --type=mount --state=failed is empty
  5. The nofail path works as intended. Power the enclosure off, reboot, and confirm the host reaches multi-user with /srv/backup absent and a logged failure. This is the test that proves you have changed the failure mode rather than only widened a window
  6. **findmnt --verify is clean.** sudo findmnt --verify --verbose reports no errors for the amended entry

Prevention

  • Every fstab edit is untested until a reboot. Schedule the reboot as part of the change, on a host you can reach the console of.
  • Run sudo findmnt --verify --verbose before rebooting. It catches malformed lines, unknown filesystem types and missing mount points — the syntax class of failure — and it will not catch timing.
  • Default to nofail for anything non-essential and express real dependencies with RequiresMountsFor= in the units that have them.
  • Fix the recovery path before you need it: know whether root is locked on your images, and if it is, make sure console access and a documented GRUB edit are part of the runbook.
  • Prefer UUID= or LABEL= over /dev/sdX, and make sure the UUID is unique — a disk cloned with dd gives two devices the same UUID, and the by-uuid symlink then points at whichever udev saw last.