Serial console and SOL - the text lifeline into a host

The BMC gives you two ways to look at a host that has stopped answering SSH: a graphical remote console (KVM over IP) and a serial console. They are not equivalent, and the difference matters most in exactly the incident where you need them.

The graphical console shows you a picture of a screen. You can read a panic on it, and then you have to transcribe it by hand or attach a screenshot to the ticket. It usually needs a Java or HTML5 viewer, a licence on some vendors, and a working browser path to the BMC.

The serial console gives you text. You can select it, paste it into an incident channel, grep it, and log every byte of it to a file that outlives the session. It is also the only one that keeps working when the graphics adapter is not initialised yet, which is most of the boot.

The four layers

A serial console is not one setting. It is four, stacked, and each one covers a different slice of the boot. Configure three of the four and the console goes dark at exactly the point the missing layer owns.

The layer people forget is the first one. Firmware console redirection is off by default on a lot of hardware, so the serial console springs to life at the GRUB menu and shows nothing at all for a host that is failing during POST — which is the failure a serial console would otherwise diagnose in seconds.

Layer 2 and 3: GRUB and the kernel

Both live in /etc/default/grub on a Debian-family or pre-BLS host:

# /etc/default/grub

# Draw the menu on both the VGA console and the serial line
GRUB_TERMINAL="console serial"
GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1"

# Kernel messages to both. Order matters - see below.
GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0,115200n8"

# Remove these on a server. They hide the messages you came for.
GRUB_CMDLINE_LINUX_DEFAULT=""

# Give a human time to interrupt the menu over a slow serial line
GRUB_TIMEOUT=10

Regenerate and reboot:

# Debian family
sudo update-grub

# RHEL family - see linux-bootloader-grub for the -o path,
# and use grubby on RHEL 8+ where BLS entries hold the arguments
sudo grubby --update-kernel=ALL --args="console=tty0 console=ttyS0,115200n8"
sudo grubby --info=DEFAULT

console= can be repeated, and the last one wins

This is the single most misunderstood part of the kernel command line. console= is not a single-valued option:

• Every console= you list receives kernel log output. So console=tty0 console=ttyS0,115200n8 prints the boot messages and any panic to both the VGA screen and the serial line.
• Only the last one becomes /dev/console, the device that gets the init process stdin/stdout and therefore the emergency shell, the fsck prompt, and the systemd password prompt for an encrypted root.

So the order is a decision, not a formality. Put ttyS0 last on a server and a systemd-cryptsetup password prompt appears on the serial line where an operator on SOL can answer it. Put tty0 last and that prompt appears only on a VGA screen nobody is looking at, and the host sits in the dark at 40% of its boot having asked a question.

115200n8 is speed, parity, data bits: 115200 baud, no parity, 8 data bits. It must match what the firmware and the BMC are configured for. A mismatch does not produce silence — it produces a stream of accented punctuation, which is a useful tell.

$ cat /proc/consoles
cat /proc/cmdline

ttyS0                -W- (EC  p  )    4:64
tty0                 -WU (E     )    4:1
BOOT_IMAGE=/boot/vmlinuz-6.8.0-51-generic root=UUID=b0434124-e13c-4dff-9006-5abf403266fe ro console=tty0 console=ttyS0,115200n8

Layer 4: the login prompt

A console that shows boot messages but no login: prompt is half a console. On systemd hosts you usually get the prompt for free: systemd-getty-generator reads the console device from the kernel command line and instantiates serial-getty@ttyS0.service for it at every boot.

That generator is why the recommended order is to fix the kernel command line first and only then go looking for the getty. Enabling the unit by hand on a host whose command line has no console=ttyS0 gives you a login prompt on a line nothing else is writing to.

# Did the generator do it for you?
systemctl status serial-getty@ttyS0.service

# Only if it did not, and only once console=ttyS0 is on the cmdline
sudo systemctl enable --now serial-getty@ttyS0.service

Reaching it: IPMI serial-over-LAN

Serial-over-LAN encapsulates the serial line in the IPMI session, so you get the console over the management network with no physical cable and no terminal server.

Start read-only. sol info tells you whether SOL is even enabled on this BMC and at what baud rate, and it changes nothing:

# Substitute your own values before running:
BMC=192.0.2.50

ipmitool -I lanplus -H "$BMC" -U admin -f /etc/ipmi/bmc.pw sol info

Set in progress                 : set-complete
Enabled                         : true
Force Encryption                : false
Force Authentication            : false
Privilege Level                 : ADMINISTRATOR
Character Accumulate Level (ms) : 50
Character Send Threshold        : 220
Retry Count                     : 7
Retry Interval (ms)             : 1000
Volatile Bit Rate (kbps)        : 115.2
Non-Volatile Bit Rate (kbps)    : 115.2
Payload Channel                 : 1 (0x01)
Payload Port                    : 623

Enabled: false is the answer to “why does SOL just hang”, and Bit Rate is the number that has to match the 115200 in your kernel command line.

Then open the session:

ipmitool -I lanplus -H "$BMC" -U admin -f /etc/ipmi/bmc.pw sol activate

You are now attached to the host’s serial line. To detach, type the escape sequence ~. — tilde, then full stop — at the start of a line.

Log the session, always

A console session you did not log is evidence you destroyed. The panic scrolls, the session drops, and the stack trace is gone.

# Substitute your own values before running:
BMC=192.0.2.50
HOSTNAME=node1

install -d -m 0750 /var/log/console
ipmitool -I lanplus -H "$BMC" -U admin -f /etc/ipmi/bmc.pw sol activate \
  | tee -a "/var/log/console/${HOSTNAME}-$(date -u +%Y%m%dT%H%M%SZ).log"

For a fleet, run a console concentrator instead — conserver or ipmiconsole from FreeIPMI — so every host’s serial output is captured continuously to disk on the management host, whether or not anybody is watching. That turns “the node rebooted and we do not know why” into a file with the panic in it.

Mind the permissions. Console logs contain whatever was typed on the console, and single-user maintenance frequently means root typing commands. 0750 and a dedicated group is the floor.

Verify it before you need it

The whole point of a serial console is that it works when nothing else does, which means the only useful time to test it is when everything else is working.

# Substitute your own values before running:
BMC=192.0.2.50

# 1. Kernel registered both consoles, ttyS0 is the preferred one
cat /proc/consoles

# 2. The login prompt exists
systemctl is-active serial-getty@ttyS0.service

# 3. SOL is enabled on the BMC and the baud rates agree
ipmitool -I lanplus -H "$BMC" -U admin -f /etc/ipmi/bmc.pw sol info

# 4. Nothing is holding the payload
ipmitool -I lanplus -H "$BMC" -U admin -f /etc/ipmi/bmc.pw sol deactivate

# 5. Attach, log in, run a command, detach cleanly
ipmitool -I lanplus -H "$BMC" -U admin -f /etc/ipmi/bmc.pw sol activate

Step 5 is the one that cannot be skipped, and it is the one that finds the firmware-redirection gap: reboot the host from the console session and watch. If the first thing you see is the GRUB menu rather than the POST output, firmware console redirection is off and a host failing before GRUB will show you a blank screen.

Knowledge check