Kernel rollback - getting back to the kernel that worked · Linux

A kernel upgrade is the one change on a Linux host that cannot be validated before it takes effect and cannot be undone without a second reboot. Rollback therefore has to be planned before the upgrade, not discovered afterwards.

The whole discipline rests on one idea: the kernel you are running, the kernels you have installed, and the kernel the bootloader will pick next time are three different things. Confusing them is why “I rolled it back” is so often followed by the same panic on the next reboot.

Three different kernels

uname -r                       # RUNNING - loaded at the last boot
rpm -q kernel                  # INSTALLED - every version on disk (RHEL family)
dpkg -l 'linux-image-*'        # INSTALLED (Debian family)
sudo grubby --default-kernel   # DEFAULT  - what the next boot will use
ls /boot/vmlinuz-*             # what is physically present

A host can be running 6.6.0, have 6.6.0 and 6.6.9 installed, and be set to boot 6.6.9 next time. Nothing is wrong with that state - it is exactly what a host looks like between dnf update and its reboot. It becomes a problem when someone reads uname -r, sees the old good version, and concludes the upgrade has not happened.

uname -r does not tell you what will boot

After a kernel package upgrade, uname -r still reports the old kernel until the host reboots. A validation step that checks only uname -r therefore passes on a host that is one reboot away from a new, untested kernel.

Check the default as well:

sudo grubby --default-kernel
sudo grubby --info=DEFAULT

The reverse trap is worse. After a rollback that only used a one-shot boot, uname -r reports the good old kernel and the host is healthy - and the next reboot, weeks later and during an unrelated incident, silently returns to the broken kernel.

Where boot entries live

Modern RHEL-family and many Debian-family systems use the Boot Loader Specification: one small file per kernel under /boot/loader/entries/, rather than a monolithic grub.cfg stanza.

ls /boot/loader/entries/
cat /boot/loader/entries/*-6.6.9.conf

title Rocky Linux (6.6.9-1.el9.x86_64) 9.4
version 6.6.9-1.el9.x86_64
linux /vmlinuz-6.6.9-1.el9.x86_64
initrd /initramfs-6.6.9-1.el9.x86_64.img
options root=/dev/mapper/rl-root ro crashkernel=1G-4G:192M rhgb quiet

The practical consequence: an entry can be present and selectable while the file it points at is missing or truncated. A host that boots to a grub> prompt or fails with “error: file not found” usually has an entry whose linux or initrd path no longer resolves - most often because /boot filled up during the upgrade and the initramfs was written incompletely.

df -h /boot                                    # the usual culprit
ls -l /boot/vmlinuz-* /boot/initramfs-*        # sizes near zero are the giveaway

Rolling back

One-shot: boot the old kernel exactly once

Use this when you want to test whether the new kernel is the cause without committing to anything.

# RHEL family
sudo grub2-reboot "Rocky Linux (6.6.0-1.el9.x86_64) 9.4"
sudo reboot

# Debian family - index into the submenu, then reboot once
sudo grub-reboot "1>2"
sudo reboot

Both write a one-time entry to the GRUB environment. The setting is consumed by the next boot and then forgotten, so the boot after that returns to the default.

That is the feature: if the old kernel also fails to boot, you have learnt something and nothing is left in a strange state. It is also the trap: a one-shot rollback that fixes production is not a rollback, it is a reprieve.

Persistent: change the default

Use this once you have decided the new kernel is bad.

sudo grubby --set-default /boot/vmlinuz-6.6.0-1.el9.x86_64
sudo grubby --default-kernel        # verify BEFORE rebooting
sudo reboot

On Debian family without grubby, set GRUB_DEFAULT to the entry’s identifier and regenerate the configuration:

sudo grep -E '^menuentry|^submenu' /boot/grub/grub.cfg | head
sudoedit /etc/default/grub          # GRUB_DEFAULT="gnulinux-advanced-.../gnulinux-6.6.0-..."
sudo update-grub

Verify before rebooting, every time. grubby --set-default against a path that does not exist fails quietly enough that people miss it, and the next reboot returns to the kernel you were trying to escape.

From the console, when the host will not boot

You do not get a shell to run any of the above. At the GRUB menu:

Press a key during the countdown to stop the timer.
Select Advanced options.
Choose the previous kernel version.
Boot it.

Once the host is up, make the change persistent immediately - before anything else, and before anyone is tempted to reboot again.

Do not remove the old kernel to free space in /boot

/boot filling up is the direct cause of most failed kernel upgrades, and the instinctive fix - delete old kernels - removes the rollback path at the exact moment it is most likely to be needed.

Keep at least two, ideally three, known-good kernels installed:

# RHEL family
sudo sed -i 's/^installonly_limit=.*/installonly_limit=3/' /etc/dnf/dnf.conf

# Debian family - keep the running and the previous kernel
sudo apt autoremove --purge    # only AFTER validating the new kernel

Remove old kernels after the new one has proven itself in production, never before the upgrade to make room.

Rebuild the initramfs when it is the problem

A kernel that panics with “Unable to mount root fs” or drops to an initramfs shell usually has a bad initramfs rather than a bad kernel: a missing storage driver, an absent LVM or multipath module, or a truncated image from a full /boot.

# RHEL family - rebuild for a specific kernel version
sudo dracut --force /boot/initramfs-6.6.9-1.el9.x86_64.img 6.6.9-1.el9.x86_64

# Debian family
sudo update-initramfs -u -k 6.6.9-1.el9.x86_64

# Confirm the driver you need is actually in there
sudo lsinitrd /boot/initramfs-6.6.9-1.el9.x86_64.img | grep -E 'nvme|megaraid|dm-'

Do this from a working kernel or a rescue environment, and check df -h /boot first - rebuilding into a full filesystem produces exactly the truncated image that caused the failure.

Version-lock the bad kernel

A rollback that leaves the bad version installable is temporary. The next patch run reinstalls it, usually unattended, usually at the weekend.

# RHEL family
sudo dnf versionlock add 'kernel-6.6.9*'
sudo dnf versionlock list

# Debian family
sudo apt-mark hold linux-image-6.6.9-1-amd64
apt-mark showhold

Record why the lock exists and who removes it. An undocumented hold is discovered eighteen months later by someone auditing why a host is four kernels behind.

The rollback plan, written before the upgrade

Before the upgrade
  uname -r                    -> record the known-good version
  grubby --default-kernel     -> record the current default
  rpm -q kernel               -> confirm at least 2 kernels installed
  df -h /boot                 -> at least 200 MB free
  installonly_limit >= 3

After the upgrade, before reboot
  grubby --default-kernel     -> is it the NEW kernel, as intended?
  ls -l /boot/vmlinuz-* /boot/initramfs-*  -> non-zero sizes

After the reboot
  uname -r                    -> the NEW version is running
  systemctl --failed          -> empty
  dmesg -l err,crit           -> nothing new
  storage, network and the application all verified

If it fails
  console -> Advanced options -> previous kernel
  grubby --set-default /boot/vmlinuz-<good>
  grubby --default-kernel     -> verify
  dnf versionlock add 'kernel-<bad>*'
  record the reason