Kernel package install - the kernel upgrade workflow

A kernel upgrade is one of the most disruptive changes for a cluster node. It requires a reboot and the node is unavailable for the duration. This lesson covers the workflow and the rollback.

The dangerous part of a kernel upgrade is not the reboot. It is the gap between “the package installed successfully” and “the node can boot”. Two conditions dominate real kernel-upgrade failures, and both are visible before you reboot if you look.

Pre-flight checks

$ df -h /boot
dkms status
uname -r

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       974M  912M   0M  100% /boot

zfs/2.2.2, 6.8.0-45-generic, x86_64: installed
nvidia/550.107.02, 6.8.0-45-generic, x86_64: installed

6.8.0-45-generic

If /boot is short of space, reclaim it before installing, not during:

# Debian/Ubuntu - remove kernels older than the running one
sudo apt autoremove --purge
dpkg --list 'linux-image-*' | grep '^ii'   # confirm what is left

# RHEL family - keep the current plus two
sudo dnf remove --oldinstallonly --setopt installonly_limit=3 kernel

Install the new kernel

# Debian/Ubuntu
sudo apt update
sudo apt install linux-image-generic

# RHEL family
sudo dnf install kernel

Note that there is no reboot on those lines. Install and reboot are separate decisions, and the verification below belongs between them.

The new kernel is installed alongside the old. The old kernel remains available; if the new kernel fails to boot, the bootloader can boot the old.

Verify before rebooting

$ NEW=$(ls -1v /boot/vmlinuz-* | tail -1 | sed 's|.*/vmlinuz-||')
echo "new kernel: $NEW"
ls -l "/boot/initrd.img-$NEW"
dkms status | grep "$NEW"
sudo grub-mkconfig -o /dev/null >/dev/null && echo 'grub config generates cleanly'

new kernel: 6.8.0-51-generic
-rw-r--r-- 1 root root 78234112 Aug 11 09:14 /boot/initrd.img-6.8.0-51-generic
zfs/2.2.2, 6.8.0-51-generic, x86_64: installed
nvidia/550.107.02, 6.8.0-51-generic, x86_64: installed
grub config generates cleanly

Reboot and verify the new kernel

Once the three gates above pass, reboot:

sudo reboot

# Substitute your own values before running:
SERVICE=nginx

# After reboot
uname -r
# Should show the new kernel version

# Verify critical services
sudo systemctl status "$SERVICE"

# Verify network
ip addr show

# Verify storage
lsblk

# Verify the out-of-tree modules actually loaded
lsmod | grep -E 'zfs|nvidia'
dkms status

Reboot requirement

A new kernel requires a reboot to take effect. Until the reboot:

• The old kernel is running.
• The new kernel is installed but not loaded.
• On next reboot, the bootloader picks the new kernel (often the default).

For clusters, the reboot is the key step. The node is unavailable during the reboot.

Have a rollback plan

The old kernel is still installed, so the rollback is a boot selection rather than a reinstall. Which command you use depends on whether the host is still reachable.

Check the prerequisite before you need it. Both commands below write to grubenv, and GRUB only reads grubenv when GRUB_DEFAULT=saved is set in /etc/default/grub and grub.cfg has been regenerated:

grep '^GRUB_DEFAULT' /etc/default/grub     # want: GRUB_DEFAULT=saved
sudo grub-editenv list                     # grub2-editenv on RHEL

On a host still shipping the distribution default of GRUB_DEFAULT=0 the rollback command succeeds, prints nothing and changes nothing.

If the host boots but the new kernel is misbehaving, take the previous kernel once:

# Substitute your own values before running:
MENU_ENTRY=1

# RHEL family - list the entries, then set the NEXT boot only
sudo grubby --info=ALL | grep -E '^(index|title)'
sudo grub2-reboot "$MENU_ENTRY"
sudo reboot

# Debian family
sudo grub-reboot "1>2"      # submenu index > entry index
sudo reboot

If the rollback is meant to stick until a fixed kernel ships, write the persistent default explicitly and verify it:

# Substitute your own values before running:
OLD_VERSION=6.8.0-45-generic

sudo grubby --set-default /boot/vmlinuz-"$OLD_VERSION"
grubby --default-kernel

If the new kernel panics before userspace, neither command is available: the host is not reachable. That case needs the out-of-band console - power-cycle from the BMC, interrupt the GRUB menu, and select the previous entry by hand.

Knowledge check