critical riskcluster affecting~240 min

Runbook: Perform a rolling kernel update

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· The target kernel version has been running in staging for at least the estates soak period, on the same hardware or hypervisor generation
· Every host in scope still has a bootable fallback kernel entry - verified per host, not assumed from policy
· The bootloader is configured so that a persistent default can be set, and that mechanism has been tested on one host
· Out-of-band console access exists for every host and has been tested on at least one
· /boot has room for the new kernel and initramfs on every host, checked before the transaction
· Third-party kernel modules in use are enumerated, and their availability for the target kernel is confirmed
· The batch size respects quorum for any clustered host in scope
· The hold duration between waves is agreed and it is measured in hours or days, not minutes

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Enumerate hosts, current kernels, fallback entries and out-of-tree modules
2Install the new kernel WITHOUT rebooting, and verify the initramfs was built
3Verify the fallback: the currently running kernel must still have a bootable entry after the install
4Set the new kernel as the boot default deliberately, and confirm the setting took
5Reboot wave 0 - staging - and validate under synthetic load
6Reboot wave 1 - one production canary per tier - drain, reboot, validate, return
7Hold wave 1 under real traffic for the agreed duration
8Gate: review kernel logs, performance, third-party modules and hardware-specific behaviour
9Reboot remaining waves in batches, gating each on the previous batch being back in rotation
10Verify fleet-wide: running kernel, persistent default, modules loaded, mounts present, no host drained
11Only after the full hold, remove or version-lock the superseded kernel package

4 · Verification

Confirm the procedure actually fixed the problem.

✓Every host reports the new kernel from uname -r, and its boot time is later than the window start
✓The persistent boot default on every host matches the kernel it is running - the host will boot the same kernel next time
✓Every out-of-tree module in use before the update is loaded after it
✓Every filesystem in fstab is mounted, and every service that was running before is running now
✓The health check passes against a real request on each host before it returns to rotation
✓No host is left drained, and pool membership matches the host count
✓Performance metrics over the hold are within the agreed threshold of the pre-update baseline

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶Rollback is booting the previous kernel, and it is only available while that kernel entry still exists - which is why removing the old package is the LAST step, after the hold
↶Set the PERSISTENT boot default back to the known-good kernel; a one-shot next-boot setting is not a rollback
↶Reboot, then confirm both the running kernel and the persistent default are the old one
↶Version-lock or remove the bad kernel package so it cannot be selected again by the next patch cycle
↶POINT OF NO RETURN: once the previous kernel package is removed from a host, that host has no fallback and recovery requires rescue media or a rebuild
↶If a third-party module fails to build against the new kernel, rollback is the only option until a compatible build exists - a forward fix is not available on that host
↶Roll back one batch at a time and restore capacity before investigating

6 · Escalation

When the runbook isn't enough, contact:

· Escalate immediately if a host does not boot the new kernel - console access, not a second reboot
· Escalate if two hosts show the same regression; the fault is in the kernel or in a shared configuration and every remaining host has it
· Escalate to the vendor or platform team if an out-of-tree module fails to load - do not disable the module to make the boot succeed without recording what it was protecting
· Escalate to the service owner before continuing past any wave that showed a performance regression, however small
· Escalate to the storage or hardware owner if a host loses a device or a network interface name after the update

A rolling kernel update is not a rolling reboot with an extra package. It is the change where the thing you are replacing is the thing that makes the host able to boot at all, where the failure mode is a machine that does not come back, and where the recovery path runs through a console rather than through Ansible.

Treat it accordingly. This runbook is deliberately slower than the rolling reboot runbook it builds on, and the extra steps are all about one question: if this kernel is bad, can this host still boot the old one?

When to use this runbook

A kernel CVE requires an update across the fleet.
A hardware or driver requirement needs a newer kernel.
The estate’s kernel baseline is being moved forward.

Not for a routine reboot. If the kernel is not changing, use the rolling reboot runbook - it has fewer steps because it has fewer ways to leave a host unbootable.

Blast radius

Potentially every host, and unlike a deployment the failure is not “the service is degraded” but “the machine is gone until someone opens a console”.

wave 0   staging          6 hosts     soak 48h
wave 1   canary           3 hosts     1 per tier, hold 24h under real traffic
wave 2   web              40 hosts    batch 4
wave 3   app              60 hosts    batch 6
wave 4   db replicas      8 hosts     batch 1
wave 5   db primaries     4 hosts     batch 1, with failover

The hold durations are the part that gets compressed under schedule pressure and the part that catches the regressions that matter. A driver fault under sustained load does not appear in the ten minutes after a reboot.

Step 1: Enumerate the starting state

Read-only / Safewhat are we starting from

ansible kernel_scope -m command -a 'uname -r' -o | tee before-kernel.txt
ansible kernel_scope -b -m command -a 'uptime -s' -o | tee before-boottime.txt
ansible kernel_scope -b -m command -a 'df -h /boot' -o | tee before-boot-space.txt

# Out-of-tree modules: these are the ones that will not exist for the new kernel
ansible kernel_scope -b -m shell \
-a 'cat /proc/modules | cut -d" " -f1 | sort' -o > before-modules.txt

Read-only / Safewhich modules are out-of-tree

ansible kernel_scope -b -m command \
-a 'grep -c "" /sys/module/nvidia/version' -o 2>/dev/null || true
ansible kernel_scope -b -m command -a 'dkms status' -o

Step 2: Install the kernel without rebooting

Configuration changeinstall, do not reboot

- name: Stage the new kernel
hosts: "{{ kernel_wave }}"
become: true
tasks:
  - name: Enough space in /boot for a new kernel and initramfs
    ansible.builtin.assert:
      that:
        - ansible_facts['mounts'] | selectattr('mount', 'equalto', '/boot')
          | map(attribute='size_available') | first > 314572800
      fail_msg: "Less than 300 MiB free on /boot - refusing to install a kernel"

  - name: Install the pinned kernel version
    ansible.builtin.dnf:
      name: "kernel-{{ target_kernel }}"
      state: present
    when: ansible_facts['os_family'] == 'RedHat'
    register: kernel_install

  - name: Install the pinned kernel version
    ansible.builtin.apt:
      name: "linux-image-{{ target_kernel }}"
      state: present
    when: ansible_facts['os_family'] == 'Debian'
    register: kernel_install_deb

Installing and rebooting are separate steps here on purpose. The install is reversible, the reboot is not, and putting a verification gate between them is the whole design.

The /boot assert is a refusal. A kernel install that fills /boot leaves a partially written initramfs, and a host with a partially written initramfs for the kernel it is about to boot is a host that will not boot.

Read-only / Safeverify the initramfs exists and is not truncated

ansible kernel_scope -b -m command \
-a 'ls -l /boot/initramfs-{{ target_kernel }}.img /boot/vmlinuz-{{ target_kernel }}' -o
ansible kernel_scope -b -m command -a 'df -h /boot' -o

Step 3: Prove the fallback still exists

This is the gate that makes rollback possible, and it is a gate rather than a note: if it fails, the change stops.

Read-only / Safethe running kernel must still be bootable

ansible kernel_scope -b -m shell -o -a '
running=$(uname -r)
if grubby --info=ALL 2>/dev/null | grep -q "vmlinuz-$running"; then
echo "FALLBACK OK: $running"
else
echo "ABORT: no bootable entry for $running"
exit 1
fi'

Step 4: Set the boot default deliberately

Configuration changeset the persistent default

ansible kernel_scope -b -m command \
-a 'grubby --set-default /boot/vmlinuz-{{ target_kernel }}' -o

# Confirm it took - do not assume
ansible kernel_scope -b -m command -a 'grubby --default-kernel' -o

Do not rely on the package manager having made the new kernel the default. Set it, then read it back. The read-back is what catches a host whose GRUB_DEFAULT is not saved, where the setting is accepted and has no effect.

Read-only / Safethe precondition for any grub default to work

ansible kernel_scope -b -m command \
-a 'grep -E "^GRUB_DEFAULT=" /etc/default/grub' -o

GRUB_DEFAULT=saved is what makes grubby --set-default and grub2-set-default meaningful. Without it GRUB ignores the saved entry and both the update and the rollback become no-ops that report success.

Step 5: Wave 0 - staging, under load

Service impact possiblereboot staging

ansible-playbook -i inventories/staging kernel-reboot.yml --list-hosts
ansible-playbook -i inventories/staging kernel-reboot.yml \
-e target_kernel=6.12.0-55.el9 --diff | tee "kernel-wave0.log"

Then put load on it. A kernel regression in the network stack, the scheduler or a storage driver is a load-dependent fault; an idle staging host proves the kernel boots and almost nothing else.

Soak for the estate’s agreed period. This is the wave where the soak is cheap.

Step 6: Wave 1 - one canary per tier

The reboot play is the rolling reboot play with kernel-specific verification added:

Service impact possiblekernel-reboot.yml

- name: Rolling kernel reboot
hosts: "{{ kernel_wave }}"
become: true
serial: "{{ kernel_batch | default(1) }}"
max_fail_percentage: 0
tasks:
  - name: Record what we are leaving
    ansible.builtin.command: uname -r
    register: kernel_before
    changed_when: false

  - name: Record the boot time
    ansible.builtin.command: uptime -s
    register: boot_before
    changed_when: false

  - name: Drain from the load balancer
    ansible.builtin.uri:
      url: "https://lb.example.com/api/pool/web/{{ inventory_hostname }}/drain"
      method: POST
      status_code: [200, 204]
    delegate_to: localhost

  - name: Allow in-flight connections to finish
    ansible.builtin.wait_for:
      timeout: 60
    delegate_to: localhost

  - name: Reboot into the new kernel
    ansible.builtin.reboot:
      msg: "Kernel update to {{ target_kernel }}"
      pre_reboot_delay: 5
      post_reboot_delay: 60
      reboot_timeout: 1800
      test_command: systemctl is-system-running --wait

  - name: The host actually rebooted
    ansible.builtin.command: uptime -s
    register: boot_after
    changed_when: false
    failed_when: boot_after.stdout == boot_before.stdout

  - name: The host is running the intended kernel
    ansible.builtin.command: uname -r
    register: kernel_after
    changed_when: false
    failed_when: target_kernel not in kernel_after.stdout

  - name: The host will boot the same kernel again next time
    ansible.builtin.command: grubby --default-kernel
    register: default_kernel
    changed_when: false
    failed_when: target_kernel not in default_kernel.stdout

  - name: Every fstab entry is mounted
    ansible.builtin.command: findmnt --verify
    changed_when: false

  - name: Read the modules the new kernel actually loaded
    ansible.builtin.command: cut -d' ' -f1 /proc/modules
    register: loaded_modules
    changed_when: false

  - name: Out-of-tree modules are loaded
    ansible.builtin.assert:
      that: item in loaded_modules.stdout_lines
      fail_msg: "Module {{ item }} is not loaded after the kernel update"
    loop: "{{ required_out_of_tree_modules | default([]) }}"

  - name: Service answers a real request
    ansible.builtin.uri:
      url: "http://{{ ansible_host }}:8080/healthz"
      status_code: 200
      return_content: true
    register: health
    retries: 18
    delay: 10
    until: health.status == 200 and 'ok' in health.content

  - name: Return to the load balancer pool
    ansible.builtin.uri:
      url: "https://lb.example.com/api/pool/web/{{ inventory_hostname }}/enable"
      method: POST
      status_code: [200, 204]
    delegate_to: localhost

  - name: Confirm it is carrying traffic before the next host drains
    ansible.builtin.uri:
      url: "https://lb.example.com/api/pool/web/{{ inventory_hostname }}"
      return_content: true
    register: pool
    retries: 6
    delay: 10
    until: "'active' in pool.content"
    delegate_to: localhost

Three verifications here do not appear in an ordinary rolling reboot:

uname -r must contain the target version. A host that rebooted onto the old kernel because the default did not take looks completely healthy otherwise.
grubby --default-kernel must also contain it. Running the right kernel and being configured to boot the wrong one is a time bomb that goes off at the next unrelated reboot.
The out-of-tree module assertion. This is the one that catches a storage or security module that failed to build, before the host takes traffic.

Step 7: Hold, then gate

Hold wave 1 under real traffic for the agreed duration. Watch:

Read-only / Safewhat to watch during the hold

# Kernel-level errors since the reboot
ansible canary -b -m command \
-a 'journalctl -k --since "1 hour ago" --no-pager -p warning' -o

# Latency and error rate against the pre-update baseline
curl -sS 'http://metrics.example.com/api/v1/query?query=p95_latency' 

# Memory and I/O behaviour - the classic kernel regression signals
ansible canary -m command -a 'vmstat 1 5' -o

The gate before widening:

No kernel warnings or errors in the log that were not there before.
Performance within the agreed threshold of the baseline, measured rather than eyeballed.
Every out-of-tree module loaded and functioning.
No hardware surprises: interface names unchanged, all block devices present.

Any one of those failing stops the rollout. A small performance regression on one canary becomes a large one across sixty hosts, and it is much harder to attribute to the kernel a week later.

Step 8: Remaining waves

Same play, larger kernel_wave, batch sizes from the blast-radius table. Run --list-hosts before each wave. Gate between waves the same way you gated after wave 1.

Clustered hosts get serial: 1 and a batch size derived from quorum, not from a percentage. Rebooting two of five quorum members simultaneously is a cluster outage caused entirely by the maintenance.

Step 9: Fleet-wide verification

Read-only / Safeprove the whole fleet

ansible kernel_scope -m command -a 'uname -r' -o | tee after-kernel.txt
ansible kernel_scope -b -m command -a 'grubby --default-kernel' -o | tee after-default.txt
ansible kernel_scope -b -m command -a 'uptime -s' -o | tee after-boottime.txt

# Every host on the new kernel?
grep -vc "$TARGET_KERNEL" after-kernel.txt

# Running kernel and persistent default agree, per host?
diff <(sort after-kernel.txt) <(sort after-default.txt) | head

# Anything left drained?
curl -sS https://lb.example.com/api/pool/web | grep -c drain

The second check is the one that is easy to skip and expensive to miss. A host running the new kernel with the old one as its persistent default will silently revert at the next reboot - a fence, a power event, next month’s patch window - with no change record to correlate against.

Step 10: Only now, retire the old kernel

Destructiveretire the superseded kernel

# Preferred: keep it installed but stop it being selected
ansible kernel_scope -b -m command \
-a 'dnf versionlock add kernel-{{ target_kernel }}' -o

# Only if /boot space requires removal
ansible kernel_scope -b -m dnf \
-a 'name=kernel-{{ old_kernel }} state=absent' -o
ansible kernel_scope -b -m command -a 'grubby --info=ALL' -o

Rollback

Rollback is booting the previous kernel. It works only while that kernel entry still exists, which is why Step 10 is last.

Service impact possibleroll a host back

HOST=web01.example.com
OLD=6.11.0-49.el9

# 1. Set the PERSISTENT default back
ansible "$HOST" -b -m command -a "grubby --set-default /boot/vmlinuz-$OLD" -o
ansible "$HOST" -b -m command -a 'grubby --default-kernel' -o

# 2. Drain, then reboot
ansible-playbook -i inventories/production kernel-reboot.yml \
--limit "$HOST" -e "target_kernel=$OLD"

# 3. Confirm running kernel AND persistent default are both the old one
ansible "$HOST" -m command -a 'uname -r' -o
ansible "$HOST" -b -m command -a 'grubby --default-kernel' -o

# 4. Stop the bad kernel being reselected by the next patch cycle
ansible "$HOST" -b -m command -a "dnf versionlock add kernel-$OLD" -o

Roll back one batch at a time and restore capacity before investigating. And record which hosts were rolled back: a fleet split between two kernels is a state that has to be closed out, not left.

Common patterns

Symptom	Likely cause	Resolution
Host does not boot at all	Bad initramfs, or `/boot` filled during install	Console; boot the fallback entry; rebuild initramfs
Host boots but storage is missing	Out-of-tree storage module did not build	Roll back; no forward fix until a compatible build exists
Host rebooted onto the old kernel	The default did not take; `GRUB_DEFAULT` is not `saved`	Fix `/etc/default/grub`, regenerate config, retest on one host
Running the new kernel, default is the old one	`grubby --set-default` was never run, or ran on a different entry	Set and read back; this is why Step 9 diffs the two lists
Network interface renamed after reboot	Predictable naming changed with the new kernel or driver	Console; pin the name; check the whole group before continuing
Regression appears days later	Load-dependent driver or scheduler fault	Why the hold is measured in days; roll back and report upstream
Rollback impossible on some hosts	`installonly_limit` pruned the old kernel at install time	The Step 3 gate; restore a fallback before rebooting
Cluster lost quorum	Batch size was a percentage, not a quorum calculation	Batch clustered hosts one at a time

Escalation

Escalate when:

A host does not boot. Console access, not another reboot attempt.
Two hosts show the same regression - the fault is shared.
An out-of-tree module will not load. Do not disable it to make the boot succeed without recording what it was doing.
Any performance regression appears, however small.
A device or interface name changed.

Why this matters

The kernel is the one component whose failure mode is “the host is not there any more”. Everything else in an estate degrades; this stops.

That is why the ordering in this runbook is what it is. Install without rebooting, so there is a gate between a reversible action and an irreversible one. Verify the fallback before the reboot, because the package manager may have quietly removed it. Set the persistent default and read it back, because a one-shot setting produces a rollback that undoes itself weeks later. Retire the old kernel last, because it is the rollback.

The step that gets cut under schedule pressure is the hold, and it is the step that catches the regressions worth catching: the driver that fails under sustained I/O, the scheduler change that costs 8% of your latency budget. Those do not show up in the ten minutes after a reboot, and by the time they show up across sixty hosts nobody is looking at the kernel any more.