Lab: Rolling reboot and kernel upgrade, with a rollback that works

Objective

By the end of this lab you will have upgraded the kernel on four hosts one at a time, with each host proving it came back on the new kernel before the next was touched, and you will have rolled one host back to its previous kernel entry through Ansible — not through a console, which is what makes it a procedure rather than an emergency.

Architecture

Four managed nodes, rebooted strictly one at a time.

controller
    │  serial: 1, max_fail_percentage: 0
    ├── node1  upgrade → reboot → validate → next
    ├── node2  same
    ├── node3  validation induced to fail here; the run stops
    └── node4  never touched

Requirements

A controller with ansible-core 2.21.x.
Four managed VMs with their own kernel, their own bootloader and systemd as PID 1. Every single thing in this lab is a boot behaviour. There is no container path and no simulation path. B-nested only.
SSH key access and become. The become path must survive a reboot — a become password prompted interactively will stall the play.
Out-of-band access to every node: a hypervisor console, serial console or IPMI. This is not optional. A kernel that does not boot cannot be reached over SSH, and the rollback in Task 6 assumes the host came back. Task 7 covers the case where it did not, and that case needs a console.
At least 2 GB free in /boot — or wherever the kernel images live — per node. A full /boot is the commonest kernel-upgrade failure and it fails after the package manager has started.

Scenario

A kernel CVE requires a fleet-wide upgrade. The last time this was done, somebody ran the upgrade play against the whole group and rebooted everything within ninety seconds. Two hosts did not come back — one had a full /boot, one had an out-of-tree module that failed to rebuild — and the outage lasted until somebody found the console credentials.

Your job is to do it one host at a time, with a gate that stops the run before the second failure.

Tasks

Task 1: Capture, and produce a reboot-required report

WORKDIR="$HOME/ansible-kernel-lab"
mkdir -p "$WORKDIR/reports"
cd "$WORKDIR"

inventory.yml:

fleet:
  hosts:
    node1: {ansible_host: 192.0.2.11}
    node2: {ansible_host: 192.0.2.12}
    node3: {ansible_host: 192.0.2.13}
    node4: {ansible_host: 192.0.2.14}
  vars:
    ansible_user: operator

# capture.yml
- name: Record the boot state of every host
  hosts: fleet
  become: true
  gather_facts: true

  tasks:
    - name: Read the installed kernel packages
      ansible.builtin.shell: |
        set -o pipefail
        dpkg-query -W -f='${Package}\n' 'linux-image-*' 2>/dev/null | sort
      args:
        executable: /bin/bash
      register: kernels
      changed_when: false
      failed_when: false

    - name: Read the free space in /boot
      ansible.builtin.shell: |
        set -o pipefail
        df -Pk /boot | awk 'NR==2 {print $4}'
      args:
        executable: /bin/bash
      register: bootfree
      changed_when: false

    - name: Check whether a reboot is already pending
      ansible.builtin.stat:
        path: /var/run/reboot-required
      register: rr

    - name: Read the default boot entry
      ansible.builtin.shell: |
        set -o pipefail
        grep -E '^GRUB_DEFAULT=' /etc/default/grub || echo 'GRUB_DEFAULT=0'
      args:
        executable: /bin/bash
      register: grubdefault
      changed_when: false

    - name: Write the capture
      ansible.builtin.copy:
        content: |
          host: {{ inventory_hostname }}
          captured: {{ ansible_date_time.iso8601 }}
          running_kernel: {{ ansible_facts.kernel }}
          installed_kernels: {{ kernels.stdout_lines | to_json }}
          boot_free_kb: {{ bootfree.stdout | trim }}
          reboot_pending: {{ rr.stat.exists }}
          grub_default: {{ grubdefault.stdout | trim }}
        dest: "{{ playbook_dir }}/reports/pre-kernel-{{ inventory_hostname }}.yml"
        mode: '0644'
      delegate_to: localhost
      become: false

    - name: Refuse to proceed on a host with less than 300 MB free in /boot
      ansible.builtin.assert:
        that: (bootfree.stdout | trim | int) > 307200
        fail_msg: >-
          {{ inventory_hostname }} has only {{ bootfree.stdout | trim }} KB
          free in /boot. A kernel install here will fail part-way and leave
          an unbootable initramfs. Clear old kernels first.
        success_msg: "{{ inventory_hostname }}: {{ bootfree.stdout | trim }} KB free in /boot"

Read-only / Safecontroller— Reads kernel and boot state on four nodes. Changes nothing.

$ ansible-playbook -i inventory.yml capture.yml

Task 2: Decide which hosts actually need a reboot

Rebooting on a schedule reboots hosts that do not need it. Ask instead:

# reboot-required.yml
- name: Determine which hosts require a reboot
  hosts: fleet
  become: true
  gather_facts: true

  tasks:
    - name: Debian family - the package manager's own flag
      ansible.builtin.stat:
        path: /var/run/reboot-required
      register: debian_flag

    - name: Compare the running kernel against the newest installed
      ansible.builtin.shell: |
        set -o pipefail
        running=$(uname -r)
        newest=$(ls -1 /boot/vmlinuz-* 2>/dev/null | sed 's|.*/vmlinuz-||' | sort -V | tail -1)
        echo "running=$running newest=$newest"
        [ "$running" = "$newest" ] && echo "MATCH" || echo "STALE"
      args:
        executable: /bin/bash
      register: kernelcmp
      changed_when: false

    - name: Report
      ansible.builtin.debug:
        msg: >-
          {{ inventory_hostname }}:
          flag={{ debian_flag.stat.exists }}
          {{ kernelcmp.stdout_lines | join(' ') }}

Task 3: The rolling upgrade play

# kernel-upgrade.yml
- name: Rolling kernel upgrade
  hosts: fleet
  become: true
  gather_facts: true
  serial: 1
  max_fail_percentage: 0

  vars:
    validation_command: /usr/local/bin/post-reboot-check

  tasks:
    - name: Record the kernel this host is running before the upgrade
      ansible.builtin.set_fact:
        kernel_before: "{{ ansible_facts.kernel }}"

    - name: Upgrade the kernel package
      ansible.builtin.apt:
        name: linux-image-amd64
        state: latest
        update_cache: true
      register: kernel_pkg

    - name: Confirm the new kernel image exists on disk before rebooting
      ansible.builtin.shell: |
        set -o pipefail
        ls -1 /boot/vmlinuz-* | sed 's|.*/vmlinuz-||' | sort -V | tail -1
      args:
        executable: /bin/bash
      register: newest_image
      changed_when: false

    - name: Refuse to reboot if the newest image is the one already running
      ansible.builtin.assert:
        that: newest_image.stdout | trim != kernel_before
        fail_msg: >-
          {{ inventory_hostname }}: no newer kernel on disk after the
          upgrade ({{ kernel_before }}). Rebooting would achieve nothing
          and would take the host out of service for no reason.
        success_msg: >-
          {{ inventory_hostname }}: will boot into
          {{ newest_image.stdout | trim }}
      when: kernel_pkg.changed

    - name: Reboot and wait for the host to become manageable again
      ansible.builtin.reboot:
        msg: "Kernel upgrade, rolling, host {{ inventory_hostname }}"
        pre_reboot_delay: 5
        post_reboot_delay: 20
        reboot_timeout: 900
        connect_timeout: 20
        boot_time_command: "cat /proc/sys/kernel/random/boot_id"
        test_command: "systemctl is-system-running --wait"
      when: kernel_pkg.changed

    - name: Re-gather facts, so ansible_facts.kernel is the new one
      ansible.builtin.setup:
        gather_subset: min

    - name: Assert the host came back on a different kernel
      ansible.builtin.assert:
        that: ansible_facts.kernel != kernel_before
        fail_msg: >-
          {{ inventory_hostname }} rebooted but is running
          {{ ansible_facts.kernel }}, the same kernel as before. The
          bootloader default did not change.
        success_msg: >-
          {{ inventory_hostname }}: {{ kernel_before }} ->
          {{ ansible_facts.kernel }}
      when: kernel_pkg.changed

    - name: Run the post-reboot validation
      ansible.builtin.command: "{{ validation_command }}"
      register: validation
      changed_when: false

    - name: Record the outcome
      ansible.builtin.copy:
        content: |
          host: {{ inventory_hostname }}
          kernel_before: {{ kernel_before }}
          kernel_after: {{ ansible_facts.kernel }}
          validation_rc: {{ validation.rc }}
          at: {{ lookup('pipe', 'date -Is') }}
        dest: "{{ playbook_dir }}/reports/upgraded-{{ inventory_hostname }}.yml"
        mode: '0644'
      delegate_to: localhost
      become: false

boot_time_command is the module's proof that a reboot happened

The reboot module runs boot_time_command before the reboot and again after reconnecting, and treats a changed value as proof the machine actually rebooted. The default is a who -b / uptime-derived value.

Two reasons to override it. The default has one-second granularity on some platforms, so a very fast reboot can produce the same value twice and the module waits for a reboot that already happened. And on a host where the default command is missing or behaves oddly, the module cannot tell.

cat /proc/sys/kernel/random/boot_id is a UUID regenerated by the kernel on every boot. It changes unambiguously, it needs no privileges, and it cannot collide.

The test_command is a different mechanism: it is what the module polls to decide the host is usable again. The default is whoami, which succeeds as soon as sshd accepts a connection — long before the system has finished starting services. systemctl is-system-running --wait blocks until the boot transaction completes, which is the honest definition of “back”.

Install the validation command first — it is the gate:

# setup-validation.yml
- name: Install the post-reboot validation
  hosts: fleet
  become: true
  gather_facts: false
  tasks:
    - name: Install the check
      ansible.builtin.copy:
        content: |
          #!/bin/bash
          # Post-reboot validation. Exits non-zero if the host is not
          # fit to carry traffic. Extend per service.
          set -uo pipefail
          fail=0

          systemctl is-system-running --quiet || {
            echo "system state: $(systemctl is-system-running)"; fail=1; }

          systemctl is-active --quiet ssh || systemctl is-active --quiet sshd || {
            echo "sshd not active"; fail=1; }

          ip route show default | grep -q . || { echo "no default route"; fail=1; }

          [ -f /etc/labcheck-fail ] && { echo "induced failure marker present"; fail=1; }

          exit "$fail"
        dest: /usr/local/bin/post-reboot-check
        mode: '0755'

Task 4: Run it, and watch each host prove itself

Service impact possiblecontroller— Upgrades the kernel and reboots four hosts, one at a time. Each host is out of service for the duration of its boot.

$ ansible-playbook -i inventory.yml kernel-upgrade.yml

PLAY [Rolling kernel upgrade] **************************************************
TASK [Upgrade the kernel package] **********************************************
changed: [node1]

TASK [Reboot and wait for the host to become manageable again] *****************
changed: [node1]

TASK [Assert the host came back on a different kernel] *************************
ok: [node1] => {"msg": "node1: 6.1.0-21-amd64 -> 6.1.0-23-amd64"}

TASK [Run the post-reboot validation] ******************************************
ok: [node1]

PLAY [Rolling kernel upgrade] **************************************************
TASK [Upgrade the kernel package] **********************************************
changed: [node2]
...

Illustrative output

Each host reboots, proves it booted a different kernel, and passes validation before the next host is touched. Watch the wall-clock: the serial reboot is slow, and that slowness is the safety.

Task 5: Induce a validation failure on node3

ansible -i inventory.yml node3 -b -m file \
  -a 'path=/etc/labcheck-fail state=touch mode=0644'

Reset the fleet to the previous kernel state, or simply run the upgrade again after installing a newer kernel — whichever your repository allows — and observe:

Service impact possiblecontroller— node3 reboots, comes back, fails validation, and the run stops before node4

$ ansible-playbook -i inventory.yml kernel-upgrade.yml

TASK [Run the post-reboot validation] ******************************************
fatal: [node3]: FAILED! => {"changed": false, "cmd": ["/usr/local/bin/post-reboot-check"], "rc": 1, "stdout": "induced failure marker present"}

NO MORE HOSTS LEFT *************************************************************

PLAY RECAP *********************************************************************
node1                      : ok=9    changed=2    unreachable=0    failed=0
node2                      : ok=9    changed=2    unreachable=0    failed=0
node3                      : ok=7    changed=2    unreachable=0    failed=1

Illustrative output

node4 is absent from the recap: never attempted, never rebooted, still on its old kernel and still carrying traffic. That is the gate working.

Classify the fleet, as in the canary lab:

cd "$HOME/ansible-kernel-lab"

ansible -i inventory.yml fleet -m setup -a 'gather_subset=min' \
  | grep -E '^\S+ \| |"kernel"' > reports/post-abort-kernels.txt

cat reports/post-abort-kernels.txt

Task 6: Roll node3 back to the previous kernel

This is the step that makes the whole procedure defensible. A kernel upgrade you cannot reverse from the controller is a kernel upgrade whose failure mode is a data-centre visit.

# kernel-rollback.yml
- name: Boot the previous kernel entry
  hosts: "{{ target | mandatory }}"
  become: true
  gather_facts: true

  tasks:
    - name: Load this host's pre-upgrade capture
      ansible.builtin.include_vars:
        file: "{{ playbook_dir }}/reports/pre-kernel-{{ inventory_hostname }}.yml"
        name: pre
      delegate_to: localhost
      become: false

    - name: Refuse to roll back without a recorded previous kernel
      ansible.builtin.assert:
        that:
          - pre.running_kernel is defined
          - pre.running_kernel != ansible_facts.kernel
        fail_msg: >-
          No usable pre-upgrade record for {{ inventory_hostname }}, or the
          host is already on the recorded kernel. Do not guess at a boot
          entry - use the console.

    - name: Confirm the previous kernel image is still on disk
      ansible.builtin.stat:
        path: "/boot/vmlinuz-{{ pre.running_kernel }}"
      register: oldimage

    - name: Refuse to roll back to a kernel that has been removed
      ansible.builtin.assert:
        that: oldimage.stat.exists
        fail_msg: >-
          /boot/vmlinuz-{{ pre.running_kernel }} is not present on
          {{ inventory_hostname }}. The previous kernel was removed and
          there is nothing to roll back to. Recover from the console or
          reinstall the package.

    - name: Find the GRUB menu entry id for the previous kernel
      ansible.builtin.shell: |
        set -o pipefail
        awk -F"'" '/menuentry_id_option/ {print $2}' /boot/grub/grub.cfg \
          | grep -F "{{ pre.running_kernel }}" | head -1
      args:
        executable: /bin/bash
      register: entry
      changed_when: false

    - name: Set the NEXT boot only - not the permanent default
      ansible.builtin.command: "grub-reboot '{{ entry.stdout | trim }}'"
      when: entry.stdout | trim | length > 0
      changed_when: true

    - name: Reboot into the previous kernel
      ansible.builtin.reboot:
        msg: "Rolling back to {{ pre.running_kernel }}"
        reboot_timeout: 900
        post_reboot_delay: 20
        boot_time_command: "cat /proc/sys/kernel/random/boot_id"
        test_command: "systemctl is-system-running --wait"

    - name: Re-gather facts
      ansible.builtin.setup:
        gather_subset: min

    - name: Assert the rollback landed on the recorded kernel
      ansible.builtin.assert:
        that: ansible_facts.kernel == pre.running_kernel
        fail_msg: >-
          {{ inventory_hostname }} is running {{ ansible_facts.kernel }},
          not the expected {{ pre.running_kernel }}. The grub-reboot entry
          did not take. Use the console.
        success_msg: >-
          {{ inventory_hostname }} rolled back to {{ ansible_facts.kernel }}

Service impact possiblecontroller— Sets a one-shot boot entry and reboots node3 into the previous kernel

$ ansible-playbook -i inventory.yml kernel-rollback.yml -e target=node3

grub-reboot sets the NEXT boot, not the default

grub-reboot writes a one-shot entry that GRUB consumes on the next boot and then discards. grub-set-default changes the permanent default.

Use grub-reboot for a rollback. If the rollback kernel also fails to boot, the next boot after that goes back to the normal default — which means a failed rollback does not compound into a host that boots the wrong thing forever.

It also means the rollback is not persistent: the host will boot the new kernel again on its next reboot unless you make the change permanent. That is a feature during an incident and a trap afterwards. Record it, and decide explicitly whether to pin.

grub-reboot requires GRUB_DEFAULT=saved in /etc/default/grub, which is the Debian and Ubuntu default but is not universal. Task 1 captured the value; check it before relying on this.

Task 7: Remove the induced failure and complete the fleet

cd "$HOME/ansible-kernel-lab"

ansible -i inventory.yml node3 -b -m file \
  -a 'path=/etc/labcheck-fail state=absent'

ansible -i inventory.yml node3 -b -m command -a '/usr/local/bin/post-reboot-check'

Then build the resume limit and complete the run:

{
  echo "# resume after validation failure on node3, $(date -Is)"
  echo "node3"
  echo "node4"
} > reports/resume.limit

ansible-playbook -i inventory.yml kernel-upgrade.yml \
  --limit @reports/resume.limit --list-hosts

ansible-playbook -i inventory.yml kernel-upgrade.yml \
  --limit @reports/resume.limit

What could go wrong?

What happens if a host does not come back from the reboot?

The reboot module polls until reboot_timeout expires, then fails that host. With serial: 1 and max_fail_percentage: 0 the play stops, so exactly one host is down. That is the best available outcome from a non-booting kernel and it is the reason for the batching.

Recovery is a console operation. At the GRUB menu, select the previous kernel from Advanced options, boot it, and then investigate. No amount of Ansible reaches a machine sitting at a boot prompt, which is why the Requirements ask for out-of-band access before you start.

What happens if the reboot succeeds but SSH does not come back?

test_command never succeeds and the module fails on timeout. The host is up and unreachable — the same recovery path. This is why systemctl is-system-running --wait is a better test command than whoami: it distinguishes “sshd is answering” from “the system finished booting”, and a host that boots to a degraded state is worth knowing about before the next host is touched.

What happens if /boot fills during the upgrade?

apt fails part-way, usually while generating the initramfs, and leaves a kernel package unconfigured. Do not reboot. The assertion in Task 1 exists to prevent reaching this state; if you are in it, clear space and dpkg --configure -a before anything else, and verify with update-initramfs -u -k all that every installed kernel has a complete initramfs.

What happens if you run this without serial: 1?

Every host reboots at once. On a four-host fleet that is a total outage; on a cluster with quorum it is a quorum loss that may not recover on its own. There is no situation in which a fleet-wide simultaneous kernel reboot is the right procedure, and the two-word fix is serial: 1.

Validation

reports/pre-kernel-node{1..4}.yml record a running kernel, an installed kernel list and free space in /boot, written before any change.
The /boot assertion fails the play on a node with under 300 MB free — test it by filling /boot on a scratch node if you want to see it.
Each reports/upgraded-*.yml shows kernel_before different from kernel_after.
With the induced marker, node3 reboots, comes back, and then fails validation, and node4 is absent from the recap.
kernel-rollback.yml -e target=node3 returns node3 to the kernel recorded in its capture, confirmed by a re-gathered ansible_facts.kernel.
Rebooting node3 again without another grub-reboot returns it to the new kernel — proving the one-shot semantics.
After Task 7, all four hosts report the same kernel version.

Expected Outcome

ansible-kernel-lab/
├── capture.yml, reboot-required.yml, setup-validation.yml
├── kernel-upgrade.yml, kernel-rollback.yml
├── inventory.yml
└── reports/
    ├── post-abort-kernels.txt
    ├── pre-kernel-node{1..4}.yml
    ├── resume.limit
    └── upgraded-node{1..4}.yml

Four hosts on the new kernel, one of which was rolled back and forward again, with a per-host record of the transition and a rollback procedure you have executed rather than documented.

Troubleshooting

The reboot task hangs until reboot_timeout. The host rebooted and test_command never succeeded. Try test_command: whoami to distinguish “never came back” from “came back degraded” — if whoami works and systemctl is-system-running --wait does not, a unit is failing and the host is up.

boot_time_command returns the same value after the reboot. The machine did not reboot. With /proc/sys/kernel/random/boot_id that is unambiguous; with the default it can be a granularity artefact. Check uptime directly.

The host comes back on the same kernel. The bootloader default did not change. On Debian family, update-grub runs from the kernel package’s postinst; if /boot/grub/grub.cfg has no entry for the new image, that step failed — usually because /boot was full. This is why the assertion after the upgrade compares against the newest image on disk.

grub-reboot reports error: unknown command. The command is grub2-reboot on RHEL family. Branch on ansible_facts.os_family.

grub-reboot succeeds and the host boots the new kernel anyway. GRUB_DEFAULT is not saved, so GRUB ignores the saved next-entry. grub-editenv list shows whether next_entry was written.

ansible_facts.kernel is unchanged after the reboot. You did not re-gather. Facts are collected once at the start of the play; the explicit setup task after the reboot is what refreshes them, and forgetting it produces an assertion that compares a value against itself.

A host is unreachable and the console shows a kernel panic. Boot the previous kernel from Advanced options in GRUB. Then, before doing anything else, work out why — an out-of-tree module that failed to rebuild (check dkms status) is the usual cause and will recur on every host.

Cleanup

This lab upgraded kernels and rebooted four machines. Some of it is reversible and some is not, and the cleanup says which.

Step 1. Remove the induced failure marker, on every host, unconditionally:

cd "$HOME/ansible-kernel-lab"
ansible -i inventory.yml fleet -b -m file \
  -a 'path=/etc/labcheck-fail state=absent'

Step 2. Confirm no host has a pending one-shot boot entry left over. A forgotten grub-reboot makes the next reboot — weeks later, during unrelated maintenance — land on an old kernel, and nobody will connect the two:

ansible -i inventory.yml fleet -b -m command -a 'grub-editenv list'

Any host reporting next_entry= has one pending. Clear it:

# Substitute your own values before running:
HOST=node3

ansible -i inventory.yml "$HOST" -b -m command -a 'grub-editenv - unset next_entry'
ansible -i inventory.yml "$HOST" -b -m command -a 'grub-editenv list'

Step 3. Decide what to do about the kernel itself.

Do not remove the new kernel to 'clean up'

The instinct is to purge the newly installed kernel package and return the hosts to where they started. Do not.

Removing the running kernel leaves a machine with no kernel matching uname -r, no matching modules directory, and no way to load a module until it reboots onto something else. Removing the other kernel destroys your rollback target.

The honest cleanup for a kernel lab is: leave the hosts on the new kernel, which is a normal and supported state, and record that in your notes. If you need the original state, roll the VM snapshot back — that is what the Requirements asked for.

If disk pressure genuinely requires removing old kernels, do it with the distribution’s own mechanism (apt autoremove --purge on Debian family, which knows not to remove the running kernel), never by hand, and never before the host has successfully booted the kernel you are keeping.

Step 4. Remove the validation script the lab installed:

ansible -i inventory.yml fleet -b -m file \
  -a 'path=/usr/local/bin/post-reboot-check state=absent'

Step 5. Verify the fleet is in a defined state:

ansible -i inventory.yml fleet -b -m shell \
  -a 'uname -r; systemctl is-system-running; grub-editenv list | grep -c next_entry || true'

Every host should report the same kernel, running (or a degraded you have investigated), and no pending next_entry.

Step 6. Keep the procedures and remove the working directory:

rm -rf deletes the whole lab directory, including anything you added

The next block removes ~/ansible-kernel-lab and everything under it, recursively and without confirmation. If you saved notes, an inventory you want to keep, or a playbook you extended, copy it out first — the step above does that for the lab’s own deliverables and knows nothing about yours.

The path is written in full and anchored to $HOME deliberately. A relative rm -rf run from the wrong directory is how people delete work they meant to keep, and the shell gives no second chance.

Confirm what you are about to delete before you delete it:

ls -la "$HOME/ansible-kernel-lab"

mkdir -p "$HOME/ansible-lab-deliverables/kernel"
cp -a kernel-upgrade.yml kernel-rollback.yml reports/upgraded-*.yml \
      "$HOME/ansible-lab-deliverables/kernel/"

rm -rf "$HOME/ansible-kernel-lab"

Under the hood

The reboot module is an action plugin, which means part of it runs on the controller. That is what allows it to survive the connection dropping — the controller-side half issues the reboot command, deliberately tolerates the transport dying, and then polls.

The sequence is: run boot_time_command and record the result; issue the reboot; wait pre_reboot_delay; poll for the connection to become usable again; run boot_time_command again and compare; if it differs, wait post_reboot_delay; then poll test_command until it succeeds or reboot_timeout expires.

Both halves of that matter. The boot-time comparison is what distinguishes “the machine rebooted” from “the connection blipped” — without it, a transient network failure looks identical to a reboot. The test_command poll is what distinguishes “sshd is answering” from “the machine is usable”, and the default whoami deliberately answers the weaker question because a stronger default would break on hosts without systemd.

ansible_facts are gathered once, at the start of the play, into the host’s variable dictionary. Nothing invalidates them when a host reboots, so ansible_facts.kernel after a reboot is still the value from before it. The explicit setup task is the only thing that refreshes them, and its absence produces an assertion that silently compares a value against itself and always passes.

What You Learned

/boot capacity is knowable before the upgrade, and an assertion turns the fleet’s most common kernel failure into a refusal.
Two reboot-required signals, and they disagree. The package manager’s flag answers a broad question; comparing uname -r against the newest image answers the narrow one. Use both.
boot_time_command: cat /proc/sys/kernel/random/boot_id is unambiguous proof a reboot happened, where the default can collide.
test_command: systemctl is-system-running --wait distinguishes “sshd answers” from “the machine finished booting”.
Facts do not refresh across a reboot. Without the explicit setup task, the assertion comparing before and after compares a value with itself.
grub-reboot is one-shot and grub-set-default is permanent. The one-shot form means a failed rollback does not compound, and it means the rollback is not persistent — which you must record and decide about.
serial: 1 is what caps the damage. One non-booting host instead of four, and node4 never touched.
Some cleanups do not restore. Leaving the fleet on the new kernel is the honest outcome; removing a running kernel to tidy up is a worse state than the lab created.

Objectives

Prerequisites