Emergency and rescue mode — when normal boot fails

A host that does not reach multi-user.target is one of two things: broken at the kernel/initramfs layer, or broken at the systemd layer. Both have recovery paths. Knowing the difference is the foundation of every boot-recovery runbook.

The two failure modes

flowchart LR
  FW[Firmware]
  GRUB[GRUB]
  K[Kernel + initramfs]
  S[systemd]
  M[multi-user.target]

  FW --> GRUB --> K --> S --> M
  K -.fails.-> KF[Kernel-side failure]
  S -.fails.-> SF[systemd-side failure]

• Kernel-side failure: kernel does not boot, panic, or initramfs cannot mount root. No journal is available; only GRUB rescue or live USB.
• systemd-side failure: kernel boots, initramfs mounts root, systemd starts — but reaches emergency.target instead of multi-user.target. The journal is available.

$ journalctl -xb -p err --no-pager | head

Aug  9 03:14:22 host systemd[1]: Failed to mount /data.
Aug  9 03:14:22 host systemd[1]: Dependency failed for local-fs.target.
Aug  9 03:14:22 host systemd[1]: Dependency failed for multi-user.target.
Aug  9 03:14:22 host systemd[1]: Failed to start ssh.service.
Aug  9 03:14:22 host systemd[1]: Reached target Emergency Mode.

rescue.target vs emergency.target

The systemd layer has two maintenance targets, not one, and they are not interchangeable. Almost every recovery procedure in this lesson assumes the emergency shell — which is the harsher of the two, and the one you land in when things are genuinely bad.

This is not documentation trivia — it is readable from the shipped units on any host:

systemctl cat rescue.target emergency.target | grep -E '^(Requires|After|DefaultDependencies)='
# rescue.target:    Requires=sysinit.target rescue.service
# emergency.target: Requires=emergency.service

rescue.target requiring sysinit.target is the whole story. Rescue gives you a working system minus the multi-user services: disks mounted, journal readable, vi and lvm where you expect them. Emergency gives you a root shell and a read-only root, and assumes nothing else worked.

Recovery from a corrupted /etc/fstab

The most common “cannot boot” cause: a typo in /etc/fstab that prevents a filesystem from mounting. systemd stops at emergency.target with “Failed to mount /data” in the journal.

$ # At the emergency shell:
mount -o remount,rw /
vi /etc/fstab
# Comment out or correct the bad entry
systemctl reboot

1. Connect via OOB console. SSH is down
2. Wait for emergency shell. systemd prints the failure cascade and gives you a root shell
3. Diagnose. journalctl -xb -p err shows the errors. cat /etc/fstab confirms the bad entry
4. Remount / read-write. mount -o remount,rw /
5. Edit fstab. Use vi, nano, or sed to fix the bad entry
6. Sync filesystem buffers before reboot
7. Reboot. systemctl reboot
8. Verify. Confirm the host reaches multi-user.target and the previously-failing mount is now active

Recovery from a failed fsck

On boot, systemd runs systemd-fsck@<device>.service for each filesystem with a non-zero passno. By default it runs in preen mode (fsck -a), which repairs what can be repaired safely and never asks a question. Most dirty filesystems are cleaned up this way and you never hear about it.

The boot only stops when fsck finds damage that preen mode refuses to touch. Then fsck exits asking to be run manually, and on a headless host the boot appears to hang.

$ # In the journal:
journalctl -xb -u systemd-fsck@dev-sda1.service --no-pager

Aug  9 03:14:23 host systemd-fsck[567]: /dev/sda1: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
Aug  9 03:14:23 host systemd-fsck[567]: (i) without -a or -p, fsck will prompt for input.
...

There are two separate kernel command-line knobs, and they are routinely confused with each other.

fsck.mode=force does not answer questions. It forces a full check of every filesystem on every single boot, and a genuinely corrupt filesystem still stops the boot exactly as before. The parameter that answers “yes to all” is fsck.repair=yes.

$ grep -E 'fsck' /proc/cmdline; awk '$1 !~ /^#/ && NF>=6 {print $2, $6}' /etc/fstab

/           1
/boot       2
/srv/nfs    0

Recovery from a failed LVM

If LVM volumes do not activate, the root filesystem may not be found. The initramfs may have started systemd’s LVM tools but they may be missing or misconfigured.

$ # At the emergency shell:
lvm vgscan
lvm vgchange -ay
mount /dev/vg0/root /sysroot
systemctl reboot

Recovery from a failed RAID

$ # At the emergency shell:
cat /proc/mdstat
mdadm --assemble --scan

The diagnostic flowchart

flowchart TD
  Start[Host not coming up]
  OOB[Connect via OOB console]
  KP[Kernel panic visible?]
  IF[Initramfs error?]
  EM[Reached Emergency Mode?]
  Res[Reached Rescue Mode?]
  RC[Reached multi-user but service down?]

  Start --> OOB
  OOB --> KP
  KP -->|Yes| KF[Use GRUB rescue or live USB]
  KP -->|No| IF
  IF -->|Yes| IF2[Use live USB or chroot]
  IF -->|No| EM
  EM -->|Yes| EDF[Journalctl -xb -p err / Fix fstab, fsck, LVM, RAID]
  EM -->|No| Res
  Res -->|Yes| RD[Single-user mode / repair and reboot]
  Res -->|No| RC
  RC -->|Service| SVC[systemctl restart or investigate unit]

Knowledge check