Kernel and host software upgrades

Why this matters in production

A kernel upgrade changes the layer Ceph’s daemons run on, and several things Ceph depends on live there.

What the kernel affects

uname -r
kubectl get nodes -o wide 2>/dev/null   # if Kubernetes clients exist

Hosts running only OSDs are less sensitive than hosts also acting as krbd or CephFS clients, where feature support matters directly.

Planning

Before upgrading:
  read the kernel changelog for block, network, and ceph subsystems
  verify the new kernel is available and bootable
  confirm the previous kernel remains installed as a fallback
  test on one host before the fleet
  plan the sequence: one host at a time, health between each

# the fallback must exist
ls /boot/vmlinuz-*
grubby --default-kernel 2>/dev/null || grep -E '^default' /boot/grub2/grub.cfg

Executing

# per host
ceph -s | grep HEALTH_OK || exit 1
ceph orch host maintenance enter ceph-03

ssh ceph-03 'dnf update kernel -y && systemctl reboot'

# wait for it to return
while ! ssh ceph-03 uname -r 2>/dev/null; do sleep 15; done
ssh ceph-03 uname -r

ceph orch host maintenance exit ceph-03

# wait for health before the next host
while ! ceph health | grep -q HEALTH_OK; do sleep 30; done

Verifying

# the OSDs are up and performing
ceph osd tree | grep -A15 ceph-03
for osd in $(ceph osd ls-tree ceph-03); do
  printf 'osd.%-4s ' "$osd"
  ceph tell osd.$osd bench 12288000 4096 4194304 100 2>/dev/null | \
    python3 -c 'import sys,json; print(round(json.load(sys.stdin)["iops"],1))'
done

# network performance unchanged
# cluster-network address of a peer Ceph node:
PEER=192.0.2.12

iperf3 -c "$PEER" -t 20 -P 8

# and device behaviour
iostat -x 1 5
dmesg -T | grep -iE 'error|fail' | tail -20

Comparing the benchmark against the pre-upgrade figure is what catches a regression that would otherwise appear as a gradual cluster slowdown.

Rolling back

# boot the previous kernel
# the release that was running before the upgrade, from ls /boot/vmlinuz-*
PREV_KERNEL=5.14.0-503.15.1.el9_5.x86_64

ssh ceph-03 "grubby --set-default /boot/vmlinuz-$PREV_KERNEL"
ssh ceph-03 systemctl reboot

The previous kernel is the rollback path, which is why it must remain
installed. A distribution that removes old kernels aggressively should be
configured not to during an upgrade campaign.

# on RHEL-family
grep installonly_limit /etc/dnf/dnf.conf

Quiz

Production discipline

Benchmark one host before and after a kernel upgrade before rolling it across the fleet — a silent throughput regression rolled to twelve hosts becomes a gradual slowdown with no attributable event. Keep the previous kernel installed; it is the rollback path.

Cross-course references

• Kubernetes: node OS upgrades are rolled with the same one-at-a-time health gating
• Linux: keeping the previous kernel bootable is the standard rollback mechanism