Runbook: Rolling kernel upgrade - the production procedure

This runbook covers a rolling kernel upgrade on a cluster node by node. The procedure is the production discipline for kernel maintenance.

When to use this runbook

Use this runbook when:

• A security update requires a kernel upgrade.
• A new kernel feature is needed.
• A driver or hardware change requires a newer kernel.
• The new kernel has been tested in staging.

Inputs

Gather before starting:

• The target kernel version.
• The cluster name and node names.
• Out-of-band access (BMC, console).
• A maintenance window (typically 24+ hours per node).
• Communication channels.

Procedure

Step 1: Capture baseline

# Current kernel
uname -r > /tmp/kernel-before.txt

# Cluster status
sudo pcs status > /tmp/cluster-before.txt

# Service health
curl -sf http://<vip>/health > /tmp/health-before.txt

Step 2: Configure GRUB for fallback

# Note the current default kernel
sudo grubby --default-kernel
sudo grub2-editenv list | grep saved_entry

# List every bootable kernel entry
sudo grubby --info=ALL | grep -E '^(index|kernel|title)='

# Explicit gate: the currently RUNNING kernel must still have an entry
running=$(uname -r)
sudo grubby --info=ALL | grep -q "vmlinuz-${running}" \
&& echo "FALLBACK OK: ${running} is still bootable" \
|| { echo "ABORT: no fallback entry for ${running}"; exit 1; }

The old kernel is the safety net. Confirm it is still in the menu before proceeding. This is a gate, not a note: if the ABORT branch fires, stop the change. dnf prunes old kernels once installonly_limit is reached, so a node that has been patched several times may have no fallback left.

Step 3: Drain node1 (canary)

sudo pcs node standby node1 --wait=300
sudo pcs status nodes

Verify resources have migrated. Node1 must appear under Standby, and every resource must be running on node2 or node3.

Standby is a node attribute in the CIB. It survives the reboot. Node1 will come back as Online (standby) and will host nothing until you clear it in Step 7. That is the point of the drain, and it is also the trap: nothing clears standby for you.

Step 4: Install the new kernel on node1

sudo dnf install kernel

Step 5: Reboot node1

sudo reboot

Wait for the host to come back. SSH in and verify.

Step 6: Validate node1

uname -r
# New kernel

sudo pcs status nodes
# Node1 rejoined the cluster, still listed under Standby

# Cluster-level health, served by node2 and node3
curl -sf http://<vip>/health

# Latency
time curl -sf http://<vip>/

# Logs
sudo journalctl -k --since "5 minutes ago" | grep -i error

Node1 is back in the membership but still drained, so these checks prove the new kernel boots and the cluster stack starts. They do not yet prove node1 can carry the workload. That is Step 7.

If validation fails, roll back:

# 1. Identify the known-good kernel entry
sudo grubby --info=ALL | grep -E '^(index|kernel|title)='

# 2. Make the OLD kernel the PERSISTENT default (survives every future boot)
sudo grubby --set-default /boot/vmlinuz-<old-version>
sudo grubby --default-kernel        # verify it took

# 3. Reboot
sudo reboot

# 4. After boot, confirm running == intended
uname -r
sudo grubby --default-kernel

# 5. Prevent recurrence: the bad kernel must not be reselectable
sudo dnf remove kernel-core-<bad-version>
# or, if it must stay installed:
sudo dnf versionlock add kernel-<good-version>

Verify the rollback. Leave node1 in standby while it is rolling back, then clear standby in Step 7 once the old kernel has booted and validated. A rolled-back node still has to go back into service.

Step 7: Return node1 to service

sudo pcs node unstandby node1 --wait=300
sudo pcs status nodes
# node1 must be listed under Online, and under no Standby line

sudo pcs status
# Resources placed on node1

The node is only upgraded when it is carrying resources again. Until unstandby runs, node1 is a spare that costs capacity and gives nothing back, and the cluster is running on the remaining nodes with no headroom.

Do not proceed to node2 until node1 is out of standby. If you drain node2 while node1 is still drained, a three-node cluster is down to one node, and the third drain takes the service down completely.

Step 8: Hold for 24 hours

After node1 is back in service, hold for 24 hours. Monitor:

• Error rates.
• Performance.
• Health check status.

The hold only counts while node1 is carrying live traffic. A drained node under no load proves nothing.

If any regression, roll back immediately.

Step 9: Update node2 and node3

Repeat the full procedure for node2 and node3, one at a time, with 24-hour holds. The full procedure is: drain, install, reboot, validate, unstandby, hold. Each node returns to service before the next is drained.

Step 10: Final validation

After all nodes updated:

# All nodes online, no node left in standby
sudo pcs status nodes

# All resources running and spread across the nodes
sudo pcs resource status

# Synthetic monitoring
curl -sf http://<vip>/health

The Standby line in pcs status nodes must be empty. If a node name is still on it, that node is online, patched, and idle, and the cluster has less capacity than the change record claims.

Step 11: Document

In the change log:

• Time of upgrade.
• Old and new kernel versions.
• Nodes updated.
• Validation results.
• Confirmation that no node was left in standby.
• Confirmation that each node’s persistent default kernel (grubby --default-kernel) matches the kernel it is running.
• Rollback tested.

Common patterns

Knowledge check