Reboot and boot validation - the kernel upgrade verification

After a kernel reboot, validate that the new kernel is running and the system is fully functional. A kernel regression can break critical features; validation catches it before the next node.

Verify the new kernel

# Check kernel version
uname -r
# Should match the new kernel package

# Check uptime
uptime
# Should be recent

# Check kernel logs
sudo journalctl -k
# Look for errors, warnings, or stack traces

Validate critical services

# Substitute your own values before running:
GATEWAY=192.0.2.1
SERVICE=nginx

# Network
ip addr show
ping -c 3 "$GATEWAY"

# Storage
lsblk
df -h

# Critical services
sudo systemctl status "$SERVICE"

# Logs
sudo journalctl -u "$SERVICE" --since "10 minutes ago" | grep -i error

If any of these fails, the kernel has a regression.

Common kernel regressions

Watch for these in the post-reboot validation:

• Network interface renamed: predictable interface names fail.
• Kernel module missing: a driver was not built.
• Storage driver issue: NVMe or RAID regression.
• cgroup v2 issue: container or systemd unit fails.
• Security policy change: SELinux or AppArmor denies previously-allowed access.

For each, the rollback is to boot the old kernel.

Roll back if validation fails

There are two different situations, and they need different commands.

The host booted, but validation failed

The host is up and reachable. You want it to come up on the previous kernel once, so you can confirm the regression is gone, then fix forward.

# Substitute your own values before running:
MENU_ENTRY=1

# RHEL family - find the entry
sudo grubby --info=ALL | grep -E '^(index|title)'

# Set the NEXT boot only
sudo grub2-reboot "$MENU_ENTRY"
sudo reboot

# Debian family - list the menu entries with their indices
awk -F\" '/^menuentry |^submenu /{print i++ ": " $2}' /boot/grub/grub.cfg

sudo grub-reboot "1>2"      # submenu index > entry index
sudo reboot

The host does not boot at all

There is no automated rollback for this case, and pretending otherwise costs you the outage window. Both grub2-reboot and grub2-set-default run on a booted, reachable host. If the new kernel panics or hangs, the host is neither.

The honest procedure is:

1. Open the out-of-band console (iDRAC, iLO, IPMI SOL,
   cloud serial console). This is why OOB access is a
   prerequisite for kernel work, not a nice-to-have.
2. Reboot the host from the OOB power controls.
3. Interrupt GRUB at the menu and select the previous
   kernel entry by hand.
4. Once it is up, run `grub-editenv list` and fix whatever
   made the one-shot rollback unavailable.

This is also why you hold the first node for 24 hours: you find the regression on one host with a console open, not on forty hosts at once.

Validate the rollback (services work, etc.). Investigate the regression in staging.

Hold for verification

For kernel upgrades, hold for 24 hours before proceeding to the next node:

• 1 hour: catch obvious regressions.
• 24 hours: catch subtle regressions (drivers, edge cases).

During the hold, monitor the node:

• Error rates in logs.
• Performance metrics.
• Health check status.

If anything goes wrong, roll back before the next node.

Knowledge check