critical riskcluster affecting~60 min

Runbook: Recover a Linux host that fails to boot

1 · Prerequisites

Confirm every item is in place before any state change.

OOB console access (IPMI, iDRAC, iLO, or hypervisor console)
Root password for the failing host
Read access to the runbook

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Confirm the host is genuinely not booting — not just SSH being down (check OOB console)
· Identify the failure mode from the console output (kernel panic, initramfs drop, emergency mode, rescue mode, boot completes)
· Verify OOB console is responsive; reboot the BMC if necessary
· Confirm the host is not mid-update — a kernel update in progress may complete normally after several minutes

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Mode 1 — kernel panic. Capture the panic output (photo or serial capture). Reboot to the previous kernel from the GRUB menu (if available). If only one kernel, boot from live USB or virtual media and chroot to fix.
2Mode 2 — initramfs drop. From the initramfs shell, run blkid, lvm vgscan, lvm vgchange -ay, mdadm --assemble --scan to bring up storage. Mount the real root manually: mount /dev/sda2 /sysroot. exit to continue boot.
3Mode 3 — systemd emergency mode. At the root password prompt, journalctl -xb -p err. Identify the failing unit (commonly /etc/fstab, a missing LVM volume, or a failed mount). BEFORE remounting: journalctl -k -b --no-pager | grep -iE "EXT4-fs error|XFS.*(Corruption|Internal error)|Buffer I/O error|blk_update_request|medium error|Remounting filesystem read-only" — a read-only root in emergency mode is expected, a filesystem the kernel demoted mid-life is not. If that grep hits, go to Mode 3b instead of remounting. If it is clean: mount -o remount,rw /. Fix the failing unit. sync. systemctl reboot.
4Mode 3b — root filesystem demoted read-only by the kernel (errors=remount-ro). Do NOT remount read-write: writes would resume on media that is returning errors. Confirm with findmnt -o TARGET,SOURCE,FSTYPE,OPTIONS /. Capture the evidence off-host: dmesg -T | grep -iE "I/O error|medium error|EXT4-fs error|XFS" | head. Identify the device with lsblk -o NAME,SIZE,SERIAL,MOUNTPOINT and check the media with smartctl -a /dev/<dev> (nvme smart-log for NVMe). Media errors or rising pending sectors: fail the service over, replace the disk, restore from backup. A one-off metadata error with clean SMART: image or snapshot the volume, then run e2fsck -f or xfs_repair from rescue media with the filesystem unmounted. Return the host to service by reboot, with the cause recorded in the incident log.
5Mode 4 — systemd rescue mode. Same as emergency mode, but more services are available (udev, journald). The same diagnostic and fix path applies.
6Mode 5 — boot completes but service down. systemctl status <service>; journalctl -xeu <service>. Identify the failure cause and restart the service or fix its configuration.
7Mode 6 — GRUB menu does not appear. Boot from live USB or virtual media mounted via the BMC. Mount the host's root at /mnt, then /boot and (UEFI) the ESP: mount /dev/<root> /mnt; mount /dev/<boot> /mnt/boot; mount /dev/<esp> /mnt/boot/efi. Bind-mount the API filesystems or grub-install cannot probe devices: for d in dev dev/pts proc sys run; do mount --bind /$d /mnt/$d; done — and on UEFI also mount --bind /sys/firmware/efi/efivars /mnt/sys/firmware/efi/efivars, without which grub-install cannot write the NVRAM boot entry and reports success while changing nothing the firmware will read. Then chroot /mnt /bin/bash and reinstall: UEFI — grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=<distro>; BIOS — grub-install /dev/sda. Regenerate the menu: grub-mkconfig -o /boot/grub/grub.cfg (update-grub on Debian/Ubuntu, grub2-mkconfig -o /boot/grub2/grub.cfg on RHEL). On UEFI, run efibootmgr -v inside the chroot and confirm the entry exists before you leave. Exit chroot; umount -R /mnt; reboot.

4 · Verification

Confirm the procedure actually fixed the problem.

✓After a GRUB reinstall on UEFI: efibootmgr -v run inside the chroot lists the reinstalled bootloader entry, pointing at the ESP path just written — "Installation finished. No error reported." is not sufficient evidence
✓After a read-only-root incident: findmnt -o TARGET,OPTIONS / shows rw, and journalctl -k -b is free of I/O and filesystem errors on the recovered device
✓The host reaches multi-user.target (systemctl is-system-running reports running)
✓The previously failing unit is now active
✓All expected services are active (systemctl list-units --state=failed returns empty)
✓SSH is reachable
✓The OOB console shows a clean boot with no FAILED units
✓The journal from the recovery is saved for post-incident review

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If the recovery introduces a new failure, reboot to the snapshot or backup taken before the change
↶If the snapshot is unavailable, boot from live USB and restore /etc/fstab, the failing config file, or the GRUB installation from backup
↶Document the failed attempt in the incident log
↶Escalate to the application owner if the failure relates to an application configuration that you cannot safely restore

6 · Escalation

When the runbook isn't enough, contact:

· If kernel panic persists across multiple reboots and across previous kernels: escalate to the hardware vendor — this is likely a hardware fault (CPU, memory, or storage controller)
· If initramfs repeatedly cannot mount the root filesystem: escalate to the storage team — the storage layer may be at fault
· If systemd repeatedly fails on the same unit after recovery: escalate to the unit's owner — there may be an application bug
· If OOB console is unresponsive: escalate to the data center or hardware vendor — the BMC may need replacement

Symptoms

Host is not reachable via SSH.
Host does not respond to ICMP ping.
OOB console shows a non-default prompt (kernel panic, initramfs drop, emergency mode, rescue mode) or a hung boot.

Diagnosis

The first diagnostic is the OOB console. Connect and observe the boot output. The failure mode is one of:

Kernel panic — kernel prints an oops and stops.
Initramfs drop to shell — initramfs could not find the root filesystem.
systemd emergency mode — root filesystem is mounted but systemd could not reach multi-user.target.
systemd rescue mode — fewer services started than multi-user.target requires.
Boot completes but service is down — the host is up, one service did not start.
GRUB menu does not appear — firmware-level or bootloader-level failure.

Mode 3 has a variant that looks identical at the prompt and is handled differently: a root filesystem the kernel itself demoted to read-only after an I/O or metadata error (errors=remount-ro). Only the kernel log separates the two, which is why step 3 runs that grep before anything is remounted.

Two commands in this runbook report success while doing nothing useful

Both are silent failures. Both have cost real outages.

mount -o remount,rw / is the correct step in emergency mode and the wrong one after errors=remount-ro. In the second case it resumes writes on a device that is returning errors and can turn a frozen, recoverable filesystem into a corrupt one. The journalctl -k grep in step 3 is what tells the two apart — run it first, every time, and save its output off the host before you change anything.

grub-install prints “Installation finished. No error reported.” even when the chroot has no /dev, no /sys, and no ESP mounted. On UEFI it cannot then call efibootmgr to write the NVRAM entry, so the firmware still has nothing to boot. The operator reboots into the same firmware menu, concludes the disk or board is faulty, and escalates to the hardware vendor. efibootmgr -v inside the chroot is the only evidence that the reinstall actually landed.