Lab: Recover a Linux host that will not boot

This lab walks through a real boot recovery on a disposable VM. The failure is contrived (a corrupt fstab entry) but the recovery procedure applies to every “host will not boot” incident.

Objective

By the end of this lab, you can:

Inject a known boot failure.
Connect via “OOB console” (the VM console) and diagnose.
Recover using emergency mode.
Verify the recovery with a clean reboot.

Architecture

flowchart LR
  VM[Disposable VM]
  CONSOLE[VM console / OOB]
  FAIL[Inject fstab failure]
  REC[Diagnose & recover]
  VER[Verify clean boot]

  VM --> FAIL --> CONSOLE --> REC --> VER

Requirements

A disposable Linux VM (Ubuntu 24.04 or Debian 12 preferred).
The ability to access the VM console (virt-manager, VirtualBox, VMware console, or KVM serial console).
A way to snapshot or roll back the VM (so you can restart the lab after the failure).
A working way into the emergency shell. On a default Ubuntu Server install there is none — see the callout below — so settle this before you break the boot, not after.

On Ubuntu the root account is locked and sulogin will refuse you

systemd’s emergency and rescue targets hand the console to sulogin, which asks for the root password. A default Ubuntu Server install never sets one: the account is locked with ! in /etc/shadow, and sudo does not exist at this point because there is no login session. sulogin has nothing to accept, so it refuses the shell and loops. That is the real recovery dead-ending, not just the lab.

Prepare one of these on the VM before Task 2:

# Option A: give root a password
sudo passwd root

# Option B: allow a passwordless emergency shell.
# Only on consoles you have physically secured - anyone at the
# console then has root without authenticating.
sudo systemctl edit emergency.service
sudo systemctl edit rescue.service
# In each drop-in:
#   [Service]
#   Environment=SYSTEMD_SULOGIN_FORCE=1

If you are already locked out with neither prepared, the fallback is the GRUB command line: append init=/bin/bash to the kernel line, which bypasses systemd and sulogin entirely. Then mount -o remount,rw /, fix the fault, and exec /sbin/init. It works, it requires console access, and it is not something you want to discover for the first time during an incident.

Scenario

You have a fresh Linux VM. You will deliberately introduce a typo into /etc/fstab that prevents a non-root filesystem from mounting. systemd will fail to reach multi-user.target and drop to emergency mode. You will then diagnose and recover using the emergency shell.

Tasks

Task 1: Snapshot the VM

Before doing anything destructive, take a snapshot. If the recovery fails or you want to start over, restore from the snapshot.

The procedure depends on the hypervisor:

libvirt / virt-manager: VM → Manage Snapshots → Add (+).
VirtualBox: Machine → Take Snapshot.
VMware: Snapshot → Take Snapshot.
KVM with virsh: virsh snapshot-create-as <vm> pre-lab.

Task 2: Inject the failure

# Add a bad entry to /etc/fstab
echo "UUID=00000000-0000-0000-0000-000000000000 /data ext4 defaults 0 2" | sudo tee -a /etc/fstab

# Confirm the entry
cat /etc/fstab

The UUID is all zeros — no real filesystem has that UUID. On the next boot, systemd will try to mount /data, fail to find the device, and stop at emergency mode.

Task 3: Reboot and observe the failure

sudo systemctl reboot

The VM will reboot and attempt to mount /data. systemd will log “Failed to mount /data”, the dependency on local-fs.target will fail, multi-user.target will fail, and systemd will drop to emergency mode.

Task 4: Connect via OOB console

The “OOB console” for a VM is the hypervisor’s console view. Open the VM console window (virt-manager, VirtualBox, VMware console, or KVM serial console) and observe the boot.

You should see:

GRUB menu (auto-boot after timeout).
Kernel boot messages.
systemd-fsck attempts to check filesystems.
The FAILED/DEPEND cascade.
“You are in emergency mode” prompt with root password request.

Task 5: Authenticate at the emergency shell

The console prompts for the root password. Enter the one you set in Requirements. If you configured SYSTEMD_SULOGIN_FORCE=1 instead, you are dropped straight to the shell with no prompt.

If it rejects you in a loop, the root account is still locked — reboot, edit the kernel line in GRUB to add init=/bin/bash and continue from there.

The shell prompt should be # (root shell) with / mounted read-only. Confirm with mount | head — / should show ro.

Task 6: Diagnose

journalctl -xb -p err --no-pager

Look for the “Failed to mount /data” message. The dependency chain (“local-fs.target failed”, “multi-user.target failed”) is the consequence.

cat /etc/fstab

Confirm the bad entry you added in Task 2.

Task 7: Remount read-write

mount -o remount,rw /

This allows editing files in /etc/.

Task 8: Fix /etc/fstab

cp /etc/fstab /etc/fstab.broken
sed -i '/UUID=00000000-0000-0000-0000-000000000000/d' /etc/fstab
cat /etc/fstab

The bad line is removed. (In a real incident, you might comment out the line with a # rather than deleting it — preserves the original for forensics.)

Task 9: Reboot and verify

sync
systemctl reboot

The VM should boot to multi-user.target without dropping to emergency mode.

systemctl is-system-running
systemctl is-active multi-user.target

Both should report running and active.

Task 10: Document

Save the journal from Task 6 and the recovery procedure as a runbook entry:

OUTDIR=/tmp/boot-recovery-$(date +%Y%m%d-%H%M%S)
mkdir -p $OUTDIR
echo "fstab recovery procedure" > $OUTDIR/README.md
# ... document the actual commands and outputs
tar -czf $OUTDIR.tgz $OUTDIR

Validation

The lab is complete when:

The VM boots to multi-user.target without dropping to emergency mode.
/etc/fstab no longer contains the bad UUID.
The journal shows a clean boot with no FAILED units.
You have captured the emergency-mode screen and the recovery commands.

Expected outcome

A repeatable procedure for the most common “will not boot” incident: a typo in /etc/fstab. The procedure applies to every similar cause (wrong mount options, missing device, wrong filesystem type).

Troubleshooting

Cannot see the VM console — confirm the hypervisor is running and the VM is accessible. virt-manager or VirtualBox console views provide direct video; KVM requires the virsh console <vm> command.
Cannot authenticate at the emergency shell — first check whether a root password exists at all. Ubuntu Server and most cloud images ship with the root account locked (! in /etc/shadow), and sulogin refuses rather than letting you in. There is nothing to type. Recover with init=/bin/bash on the GRUB kernel line, then set passwd root or the SYSTEMD_SULOGIN_FORCE=1 drop-in so the next failure is survivable. If a password was set at installation and is being rejected, use the current value.
Cannot edit fstab — confirm / is mounted read-write with mount | head. If still read-only, the kernel command line may include ro that needs overriding.

Cleanup

# Restore the snapshot if you want a clean slate
# (procedure depends on hypervisor)

Or, if you want to keep the modifications:

# Remove the /etc/fstab.broken backup
sudo rm /etc/fstab.broken

What you learned

You can now diagnose and recover from a “host will not boot” incident using emergency mode and the journal. The discipline is to capture the console output, read the journal for the root cause, fix the file, and verify with a clean reboot.

Objectives

Prerequisites