Boot failure diagnosis from the OOB console

The OOB console output is the first diagnostic you have when a host is not coming up. Knowing how to read it — and which failure mode each pattern indicates — is the difference between a 5-minute diagnosis and a multi-hour recovery.

The four failure modes

flowchart TD
  A[Host not coming up]
  K[Kernel panic]
  I[Initramfs drop to shell]
  E[systemd emergency mode]
  R[systemd rescue mode]
  S[Boot completes but service down]
  A --> K
  A --> I
  A --> E
  A --> R
  A --> S

Each failure mode has a distinct console signature.

Mode 1: kernel panic

$ # Example kernel panic (synthetic, for illustration):

[   24.567] BUG: unable to handle page fault at ffffffc0deadbeef
[   24.567] Oops: 0010 [#1] SMP PTIT
[   24.567] CPU: 2 PID: 1234 Comm: myapp Not tainted 6.6.31 #1
[   24.567] Hardware name: Vendor Product/Board, BIOS 2.1 12/3/2024
[   24.567] task: ffff...  ti: ffff...  task.ti: ffff...
[   24.567] RIP: 0010:[<ffffffffc0deadbeef>]  [<ffffffffc0deadbeef+0x10/0x20>]
[   24.567] Code: 48 89 e5 48 83 ec 10 ...
[   24.567] RSP: 0018:ffff... EFLAGS: 00010246
[   24.567] RAX: ffffffc0deadbeef RBX: ...
...
[   24.567] Kernel panic - not syncing: Fatal exception in interrupt
[   24.567] ---[ end Kernel panic - not syncing: Fatal exception ]---

The kernel panic signature:

• “Oops:” or “BUG:” lines.
• Register dump (RAX, RBX, RIP, etc.).
• Stack trace with function names if available.
• “Kernel panic - not syncing” final line.

Recovery:

1. Capture the panic output (photo, serial capture, or — properly — a vmcore).
2. Reboot with the previous kernel from GRUB menu (if available).
3. If only one kernel, boot from live USB and chroot.
4. Report the panic with the captured output; the kernel developer can decode the trace.

Capture the crash properly: kdump

kdump reserves a slice of memory at boot, loads a second “capture” kernel into it with kexec, and hands control to that capture kernel when the production kernel panics. The capture kernel writes the memory of the dead kernel to disk as a vmcore. That file contains everything the console photograph does not: the full stack, every task, the slab state.

Reserve the memory on the kernel command line:

# /etc/default/grub — GRUB_CMDLINE_LINUX
crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M
# or, on distributions that support it:
crashkernel=auto

crashkernel= is not optional. Without a reservation there is nowhere to load the capture kernel, and kdump silently stays disabled.

$ sudo systemctl enable --now kdump.service; cat /sys/kernel/kexec_crash_loaded

1

The vmcore lands in /var/crash/<timestamp>/ by default. Size that filesystem for it: makedumpfile compresses and strips free pages, but a large-memory host still writes gigabytes. Triage the dump with crash /usr/lib/debug/.../vmlinux vmcore and start with bt, log, and ps.

Make the panic behaviour deliberate

By default a panicked kernel sits there forever. That is useful if a human is watching the console and useless at 04:00.

# /etc/sysctl.d/90-panic.conf
kernel.panic = 30          # reboot 30s after a panic
kernel.panic_on_oops = 1   # treat an oops as a panic: capture, then reboot

kernel.panic = 30 gives kdump time to write the vmcore, then brings the host back into service. kernel.panic_on_oops = 1 is the fail-fast choice: a kernel that has already oopsed has corrupt state, and a host that limps on after an oops produces worse outcomes than one that reboots cleanly. Apply with sudo sysctl --system.

Mode 2: initramfs drop to shell

$ # Example initramfs error:

[   18.234] dracut-mount: Waiting for device /dev/sda2
[   18.234] dracut-mount: no root device found
[   18.234] dracut: FATAL: Failed to mount the root filesystem
[   18.234] dracut:
[   18.234] dracut: Generating /run/initramfs/rdsosreport.txt
[   18.234] dracut:
[   18.234] You are now being dropped into an emergency shell.
[   18.234] sh: cannot access tty; job control turned off
[initramfs /]# _

Recovery from the initramfs shell:

# Identify devices
blkid
ls -l /dev/disk/by-uuid/
ls -l /dev/sd*

# Activate LVM if needed
lvm vgscan
lvm vgchange -ay

# Assemble RAID if needed
mdadm --assemble --scan

# Manually mount the root filesystem
mount /dev/sda2 /sysroot
# or
mount /dev/mapper/vg0-root /sysroot

# Exit the initramfs shell and continue boot
exit

Mode 3: systemd emergency mode

$ # systemd emergency output (synthetic):

[  OK  ] Reached target Local File Systems
[FAILED] Failed to mount /data
[DEPEND] Dependency failed for Local File Systems
[DEPEND] Dependency failed for Multi-User System
[DEPEND] Dependency failed for Graphical Interface
[ 12.345] You are in emergency mode. After logging in, type journalctl -xb to view
[ 12.345] system logs, systemctl reboot to reboot, systemctl default or exit
[ 12.345] to try to boot into default mode
[ 12.345]
Give root password for maintenance
(or press Control-D to continue):

Recovery:

# At the emergency shell prompt (after root password):
journalctl -xb -p err --no-pager

# BEFORE remounting rw: is the root filesystem read-only because
# systemd put it there, or because the kernel demoted it after an
# I/O or metadata error?
journalctl -k -b --no-pager | grep -iE \
  'EXT4-fs error|XFS.*(Corruption|Internal error)|Buffer I/O error|blk_update_request|medium error|Remounting filesystem read-only'

# No hits: the read-only root is just emergency.target. Proceed.
mount -o remount,rw /
# Fix the failing unit (commonly /etc/fstab, a missing LVM volume, etc.)
vi /etc/fstab
sync
systemctl reboot

Why that grep comes first

Emergency mode mounts / read-only on purpose, and remounting it read-write is the normal next step. A filesystem that went read-only while the host was running is a completely different event: errors=remount-ro is the ext4/XFS response to a metadata or I/O error, and it is the kernel protecting your data by refusing further writes.

Telling those two apart is the whole point of the grep. Remount read-write on a host in the second state and you resume writing to media that is returning errors, replay the journal over bad sectors, and turn a frozen but recoverable filesystem into a corrupt one.

The triage when the kernel log does show errors:

1. Confirm the state, do not assume it: findmnt -o TARGET,SOURCE,FSTYPE,OPTIONS / shows ro in the options for a demoted filesystem.
2. Read the kernel ring buffer for the first error, not the last: dmesg -T | grep -iE "I/O error|medium error|EXT4-fs error|XFS" | head. The first line names the device and the sector.
3. Identify the physical device behind the mount: lsblk -o NAME,SIZE,SERIAL,MOUNTPOINT and grep . /sys/block/<dev>/device/{vendor,model}.
4. Check the media: smartctl -a /dev/<dev> — look at Reallocated_Sector_Ct, Current_Pending_Sector, Offline_Uncorrectable, and the SMART error log. On NVMe: nvme smart-log /dev/nvme0.
5. Decide from the evidence. Media errors or a growing pending-sector count: the device is dying. Fail the service over, replace the disk, restore. Do NOT remount read-write to buy time.
6. A one-off metadata error with clean SMART: schedule an unmount and a repair (e2fsck -f or xfs_repair) from rescue media, after snapshotting or imaging the volume. Repair tools write; an image gives you a second attempt.
7. Only once the cause is understood and written into the incident log should the filesystem come back read-write — and then usually via a reboot, not a live remount.

Mode 4: systemd rescue mode

$ # systemd rescue mode output (synthetic):

[  OK  ] Reached target Basic System.
[  OK  ] Reached target Local File Systems.
[FAILED] Failed to start MyApp daemon.
[DEPEND] Dependency failed for Multi-User System.
[ 10.234] You are in rescue mode. After logging in, type "journalctl -xb" to view
[ 10.234] system logs, "systemctl reboot" to reboot, or "exit"
[ 10.234] to continue bootup.
[ 10.234]
Give root password for maintenance
(or press Control-D to continue): _

Telling rescue and emergency apart on the console

systemd prints both messages from one template — the mode name is the only word that differs:

You are in <mode> mode. After logging in, type "journalctl -xb" to view
system logs, "systemctl reboot" to reboot, or "exit"
to continue bootup.

So read the word, and read the lines above it. In rescue mode the boot log shows Reached target Local File Systems; in emergency mode it does not, because nothing beyond the root mount was attempted.

Confirm on any running host:

systemctl cat rescue.target emergency.target | grep -E '^(Requires|After|DefaultDependencies)='

Mode 5: boot completes, service is down

$ # After boot:

$ systemctl status myapp
● myapp.service - MyApp daemon
Loaded: loaded (/etc/systemd/system/myapp.service; enabled)
Active: failed (Result: exit-code) since Mon 2026-08-04 11:24:00 UTC; 5min ago
...
$ journalctl -u myapp -n 20 --no-pager

The diagnostic decision tree

flowchart TD
  Start[OOB console shows]
  K[Kernel panic?]
  I[Initramfs drop to shell?]
  E[Emergency mode prompt?]
  R[Rescue mode prompt?]
  S[Login prompt but service missing?]

  Start --> K
  K -->|Yes| K2[Capture panic / reboot to previous kernel]
  K -->|No| I
  I -->|Yes| I2[blkid / lvm / mdadm / mount / exit]
  I -->|No| E
  E -->|Yes| E2[Journalctl / fix / reboot]
  E -->|No| R
  R -->|Yes| R2[Filesystems are mounted: fault is above local-fs]
  R -->|No| S
  S -->|Yes| S2[systemctl status for the failing service]

Production discipline

The OOB console is the only reliable access when the host is not coming up. The discipline:

1. Document the OOB console access (URL, credentials) in the runbook.
2. Test OOB console access quarterly — credentials expire, certificates rotate.
3. Capture console output for every boot failure.
4. Practice recovery paths on a clone host before the real incident.
5. Update the runbook after every incident with the actual sequence that worked.

Knowledge check