Lab: Rolling kernel upgrade on a cluster

This lab performs a rolling kernel upgrade on a cluster node by node. By the end you will have a working procedure and a tested rollback.

Tasks

Task 1: Capture baseline

# Current kernel
uname -r > /tmp/kernel-before.txt

# Cluster status
sudo pcs status > /tmp/cluster-before.txt

# Service health
curl -sf http://<vip>/health > /tmp/health-before.txt

Task 2: Configure GRUB for fallback

# Note the current default
sudo grub2-editenv list | grep saved_entry

# Set the new kernel to be the one-time boot
sudo grub2-editenv list
# Check that the new kernel is set as default (after install)

Task 3: Install the new kernel

On node1 (canary):

sudo dnf install kernel

# Record the exact version. Every other node gets this one.
rpm -q kernel | tail -1 > /tmp/kernel-target.txt

The new kernel is installed alongside the old. Write the exact version down now: node2 and node3 must receive the version node1 validated, not whatever the repository holds when you get to them.

Task 4: Drain node1

sudo pcs node standby node1

# Confirm the drain finished before you touch the node.
# "standby" is a request; resource migration takes time.
sudo pcs status nodes
sudo pcs status resources | grep node1    # must return nothing

Do not proceed while any resource is still listed on node1. Draining is asynchronous. A stop that is still in progress looks identical to a drain that has completed if you only read the node state.

Know which host your shell is on

pcs node standby node1 is a cluster-wide operation. You can issue it from any member and it targets the node you name. reboot is a local operation with no node argument: it reboots the machine your shell is on.

Mixing the two in one shell is how operators reboot an in-service node while a different node sits drained. Every command from here to the end of Task 6 runs on node1. SSH there first and confirm it:

ssh node1
hostname     # must print node1 before you type anything else

Task 5: Reboot into the new kernel

On node1, and only on node1:

# Confirm the old kernel entry is still present as a fallback
sudo grubby --info=ALL | grep -E '^(index|kernel)'

sudo reboot

Wait for the host to come back. SSH in and verify:

uname -r
# Should show the new kernel

sudo pcs status
# Node1 should be a cluster member again - but still in standby

Node1 rejoins the cluster on boot, and it rejoins in standby. Standby is stored in the CIB, so it survives the reboot. The node is a member, it is not running anything, and pcs status reports this as normal.

Task 6: Validate node1 while it is still drained

Validate the node before you let it carry traffic. On node1:

# The kernel that is running must be the kernel you installed
uname -r
rpm -q kernel                     # running vs installed must agree

# Boot-time errors
sudo journalctl -k -b --priority=err

# The things kernel upgrades break: network and storage drivers
ip -br link
sudo multipath -ll                # if multipath is in use
sudo pvs && sudo vgs

# Critical services on this host
sudo systemctl --failed

If validation fails, roll back before returning the node:

sudo grubby --info=ALL             # find the old kernel index
sudo grubby --set-default-index=1  # persistent: survives every reboot
sudo reboot

Use grubby --set-default-index rather than grub2-reboot. grub2-reboot is a one-shot override: the node boots the old kernel once and then boots the bad kernel again on its next reboot, which is typically a fencing event weeks later with nobody watching.

Task 6b: Return node1 to service

This is the step that gets skipped, and skipping it is how a “successful” upgrade ends with a cluster that runs nothing.

# From any cluster member
sudo pcs node unstandby node1
sudo pcs status nodes              # node1 must appear under Online, not Standby

# Confirm it is actually carrying work again
sudo pcs status resources
sudo pcs status --full | grep -A5 -i failed    # must be empty

Only now is the service validated end to end:

curl -sf http://<vip>/health
time curl -sf http://<vip>/

Task 7: Hold for 24 hours

After node1 is back in service, hold for 24 hours. Monitor:

Error rates in logs.
Performance metrics.
Health check status.

A node in standby is not being tested. The hold only means something once the node is carrying resources again, which is the other reason Task 6b cannot be deferred.

If anything goes wrong, drain node1 again and roll it back.

Task 8: Update node2 and node3

Repeat the whole cycle for one node at a time. Never drain two nodes at once: on a 3-node cluster that leaves a single node holding everything with no failover margin, and on a 2-node cluster it loses quorum.

For each node in turn - node2, then node3:

# Substitute your own values before running:
TESTED_VERSION=6.8.0-51.52

# --- from an in-service node ---
sudo pcs node standby node2
sudo pcs status resources | grep node2     # must return nothing

# --- now SSH to node2; every command below runs ON node2 ---
ssh node2
hostname                                   # must print node2
sudo dnf install kernel-"$TESTED_VERSION"
sudo grubby --info=ALL                     # old entry still present
sudo reboot

# --- after node2 is back, from any node ---
ssh node2 uname -r                         # must be the new kernel
ssh node2 'rpm -q kernel'                  # running vs installed agree
ssh node2 'sudo journalctl -k -b --priority=err'

sudo pcs node unstandby node2              # RETURN IT TO SERVICE
sudo pcs status nodes                      # node2 under Online
sudo pcs status --full | grep -A5 -i failed  # empty before the next node

Install the exact version you validated on node1, not a bare dnf install kernel. A bare install picks up whatever the repository holds today, which may be a newer build than the one your canary proved. Then hold 24 hours and repeat for node3.

Task 9: Final validation

After all nodes are updated, the check is not “the cluster looks fine”. The check is “no node is left drained”.

# Node states - this is the assertion that matters.
# No node may be in standby or maintenance.
sudo pcs status nodes

# Same answer from the CIB, script-friendly:
sudo crm_mon -1 | grep -iE 'standby|maintenance'   # must return nothing

# All resources running, and where you expect them
sudo pcs status resources

# No failed actions anywhere
sudo pcs status --full | grep -A5 -i failed

# Synthetic monitoring through the VIP
curl -sf http://<vip>/health

# 24h monitor
journalctl -k --since "24 hours ago" --priority=err

Task 10: Document

KERNEL UPGRADE RECORD
Date: 2026-08-09
Change: kernel 5.14.0 to 5.15.0
Reason: Security update / simulated
Nodes: 3-node cluster, one at a time
Hold: 24 hours per node

Validations:
- node1: boot PASS, network PASS, storage PASS, services PASS
- node2: boot PASS, network PASS, storage PASS, services PASS
- node3: boot PASS, network PASS, storage PASS, services PASS

Returned to service (unstandby):
- node1: YES, carrying resources, verified 14:20
- node2: YES, carrying resources, verified 16:05
- node3: YES, carrying resources, verified 17:40
Standby/maintenance grep at close: no output (PASS)

Rollback tested: yes, in staging
Rollback method: grubby --set-default-index (persistent)
Rollback duration: 2 minutes per node

Result: success

The “returned to service” block is not paperwork. It is the only line in the record that distinguishes a completed upgrade from a cluster that has been quietly drained to nothing.

Objectives

Prerequisites