Rolling kernel upgrades across a fleet · Ansible

A kernel upgrade is the one change on a Linux host that cannot be validated before it takes effect and cannot be undone without a second reboot. Across a fleet, that property compounds.

Rebooting a production fleet simultaneously is not acceptable. Not “slower than ideal”, not “worth avoiding if you have time” — not acceptable, and the reasons are worth stating in full, because “it will be down for four minutes” is the argument people expect and it is the weakest one available.

Why simultaneous is different in kind

Quorum does not survive it. Anything consensus-based — etcd, a database replica set, Ceph monitors, a Consul or ZooKeeper ensemble — requires a majority of members to remain up. Reboot all of them at once and there is no majority; the cluster does not degrade, it stops. Some of these recover on their own when members return. Some require manual intervention to re-establish a quorum they lost cleanly, which is a 2 a.m. procedure nobody has practised.

Every cache starts cold at once. A service whose steady state depends on a warm cache does not return to steady state when it comes back — it returns to a cold-start state, under full production load, with no warmed peer to take the pressure. The classic outcome is a thundering herd against the database behind it, which then falls over, which prevents any host from warming. The fleet is up and the service is down.

The dependencies the fleet needs to come back are in the fleet. Hosts need DNS to resolve, LDAP or Kerberos to authenticate, NTP to agree on time, a package mirror, an artifact store, a log collector. If those run on machines you have just rebooted alongside everything else, the fleet is trying to boot into an environment that is also booting. Ordering matters and a simultaneous reboot has no ordering.

There is no healthy host left to compare against. This is the one that hurts most during the incident. Diagnosis of “is this host behaving oddly” is almost always differential: you look at a host that works and a host that does not. Reboot everything at once into a bad kernel and every host is equally broken, with nothing to diff against, and the fastest diagnostic tool you have is gone.

The loop

Per batch: drain, patch, reboot if required, reconnect, prove the new kernel is running, validate, return to service, soak. Then the next batch.

Service impact possiblethe fleet-wide preconditions, unbatched, before anything changes— Read-only in effect: gathers facts, checks disk and reachability, and refuses the whole run if any host would fail for reasons unrelated to the kernel. Runs against every host at once because the questions are fleet-wide.

- name: Pre-flight the whole fleet before any batch starts
hosts: '{{ patch_target }}'
gather_facts: true
any_errors_fatal: true
tasks:
  - name: Refuse to start with insufficient space on /boot
    ansible.builtin.assert:
      that: >-
        (ansible_facts.mounts | selectattr('mount', 'equalto', '/boot')
         | map(attribute='size_available') | first | default(0)) > 314572800
      fail_msg: >-
        {{ inventory_hostname }} has under 300 MB free on /boot. A kernel
        install here will write a truncated initramfs and the host will
        not boot.

  - name: Record the fleet-wide kernel baseline
    ansible.builtin.debug:
      msg: >-
        {{ ansible_play_hosts_all | length }} hosts, running kernels:
        {{ ansible_play_hosts_all
           | map('extract', hostvars, ['ansible_facts', 'kernel'])
           | unique | list }}
    run_once: true

any_errors_fatal: true on the pre-flight play means one host failing its disk check stops the whole run before anything is patched. That is the correct severity for a pre-flight: the point of running it is to find out now.

The run_once in that play is safe because the play is not batched. Under serial it would run once per batch — the trap the next section deals with.

Service impact possiblethe rolling play— Takes hosts out of service, upgrades the kernel, reboots, and returns them. Every host in a batch is out of service simultaneously; max_fail_percentage: 0 stops the run at the first batch that does not come back correctly.

- name: Rolling kernel upgrade
hosts: '{{ patch_target }}'
become: true
serial: '{{ patch_serial }}'
max_fail_percentage: 0
tasks:
  - name: Record the kernel this host is running before we touch it
    ansible.builtin.set_fact:
      kernel_before: '{{ ansible_facts.kernel }}'

  - name: Take this host out of service and put it back whatever happens
    block:
      - name: Drain from the load balancer
        ansible.builtin.include_tasks: drain.yml

      - name: Install the new kernel
        ansible.builtin.include_tasks: kernel-install.yml

      - name: Reboot into it
        ansible.builtin.reboot:
          msg: 'Kernel upgrade, window {{ patch_window_id }}'
          reboot_timeout: 900
          post_reboot_delay: 30
        register: reboot_result

      - name: Re-gather facts so we are describing the host that came back
        ansible.builtin.setup:
          gather_subset: ['!all', '!min', 'kernel', 'hardware']

      - name: Prove the running kernel actually changed
        ansible.builtin.assert:
          that: ansible_facts.kernel != kernel_before
          fail_msg: >-
            {{ inventory_hostname }} rebooted after
            {{ reboot_result.elapsed }}s but is still running
            {{ kernel_before }}. The bootloader default was not updated,
            or the new kernel failed and the host fell back.
          success_msg: >-
            {{ inventory_hostname }}: {{ kernel_before }} ->
            {{ ansible_facts.kernel }}

      - name: Validate the host is doing its job
        ansible.builtin.include_tasks: validate.yml

    always:
      - name: Return healthy hosts to service, announce the rest
        ansible.builtin.include_tasks: undrain.yml

  - name: Soak before the next batch
    ansible.builtin.pause:
      seconds: '{{ soak_seconds | default(120) }}'
    run_once: true

`uname -r` after the reboot is the proof

ansible_facts.kernel is the running kernel — confirmed on ansible-core 2.21.3, where a setup run reported ansible_kernel: 7.0.0-29-generic, the same string uname -r prints.

The assertion above compares it against the value recorded before the reboot, and it is the single most important task in the play, because it is the only one that distinguishes these three outcomes:

Outcome	`reboot` module	Health checks	Kernel assertion
new kernel running	pass	pass	pass
rebooted into the old kernel	pass	pass	fail
new kernel panicked, host fell back	pass	pass	fail

The middle row is the quiet one. Everything works. The service is healthy. The host is running the kernel it was running yesterday, and the vulnerability the window existed to close is still there — on a machine your patch report lists as done.

The installed kernel is not the booting kernel

A host can be running 6.6.0, have both 6.6.0 and 6.6.9 installed, and be set to boot 6.6.9 next time. That is the normal state between the package install and the reboot, and it is why “the kernel package is current” is not a claim about what is executing.

The Linux course covers the bootloader side in detail. Two facts from it matter to a rolling play:

grub2-reboot and grub-reboot set a one-shot entry. The next boot consumes it and then forgets it, so the boot after that returns to the default. That is exactly the right tool for testing a suspect kernel — if it does not come back, a power cycle returns you to the known-good default with nothing left in a strange state.

It is the wrong tool for a rollback you intend to keep. A one-shot boot that fixes production is a reprieve, not a rollback, and the next reboot — weeks later, during an unrelated incident — silently returns to the kernel that broke.

A persistent change is grubby --set-default, verified with grubby --default-kernel before rebooting. On Debian-family hosts without grubby, that means editing GRUB_DEFAULT and running update-grub.

Configuration changea canary that tests a kernel without committing to it— Sets a one-shot boot entry on the RHEL-family canary. Changes bootloader state for exactly one boot; a failure to return is recoverable by power-cycling, which lands back on the current default.

- name: Set a one-shot boot into the new kernel on the canary only
ansible.builtin.command:
  argv: [grub2-reboot, '{{ candidate_kernel_entry }}']
changed_when: true
when:
  - ansible_facts.os_family == 'RedHat'
  - inventory_hostname == ansible_play_hosts_all[0]

That when condition is the subject of the next section, and it is not run_once.

`run_once` is per batch, and that breaks fleet-wide steps

Verified on ansible-core 2.21.3: in a play with serial: [1, 3, 6] over ten hosts, a task marked run_once: true executed three times — once for each batch, on the first host of that batch.

Read-only / Saferun_once under serial, executed on 2.21.3— Ten local-connection hosts, serial: [1, 3, 6], a debug task with run_once: true. Only the debug and assert modules ran; no host state was touched.

$ ansible-playbook -i inv10.ini batchstop.yml

PLAY [Validation failure stops the next batch] **********************************
"msg": "BATCH h01 of 10 total"

PLAY [Validation failure stops the next batch] **********************************
"msg": "BATCH h02,h03,h04 of 10 total"

PLAY [Validation failure stops the next batch] **********************************
"msg": "BATCH h05,h06,h07,h08,h09,h10 of 10 total"

Three PLAY banners, three executions of a task that says “run once”.

For a soak pause, that is exactly right — one pause per batch is the intent. For anything that must genuinely happen once across the whole run, it is a bug, and in a kernel-upgrade context the candidates are real: taking a fleet-wide snapshot, opening a change ticket, silencing monitoring for the window, setting the one-shot boot entry on the canary.

Read-only / Safethe two idioms, side by side— Read-only. The first runs once per batch; the second runs once for the entire play regardless of batching, because it tests against the play-wide host list rather than the batch.

# Once per batch — correct for a soak, a batch banner, a per-batch pause.
- name: Soak before the next batch
ansible.builtin.pause:
  seconds: 120
run_once: true

# Once for the whole run — correct for anything fleet-wide.
- name: Silence monitoring for the whole window
ansible.builtin.uri:
  url: 'https://monitoring.example.com/api/v2/silences'
  method: POST
  body_format: json
  body:
    matchers: [{ name: 'group', value: '{{ patch_target }}' }]
    comment: 'Kernel window {{ patch_window_id }}'
  headers:
    Authorization: 'Bearer {{ monitoring_token }}'
delegate_to: localhost
when: inventory_hostname == ansible_play_hosts_all[0]

ansible_play_hosts_all holds every host the play targets, unaffected by batching. Testing inventory_hostname == ansible_play_hosts_all[0] is true for exactly one host in exactly one batch — the first host of the first batch — which is genuinely once per run.

Under the hood

Why the play restarts per batch, and what that means for a kernel run.

The play iterator computes the batch slices at play start from the resolved host list, then hands the strategy one slice at a time. For each slice the strategy executes the entire play — pre_tasks, roles, tasks, handler flush, post_tasks — against those hosts, and only then returns for the next slice. This is why the PLAY banner repeats, verified above.

Three consequences specific to a kernel upgrade.

Facts gathered in a batch belong to that batch. A set_fact recording kernel_before on h01 is not visible when h05’s batch runs, because that batch is a fresh execution whose active host list does not include h01. Fleet-wide reasoning — “are all hosts on the same kernel” — must happen in the separate unbatched pre-flight play, where ansible_play_hosts is the whole fleet.

Handlers flush per batch. A handler notified during batch one fires at the end of batch one, not at the end of the run. For a kernel play that is the behaviour you want: a reboot triggered by a handler must happen inside the batch that installed the kernel, not after the last batch.

The host list is frozen at play start. A host added to a dynamic inventory mid-run is never batched and never patched, and does not appear in the recap. On an autoscaling tier this means new instances launched during the window come up on whatever the image carries — usually the old kernel — and nothing in the run output hints at it.

The reboot module interacts cleanly with all of this because it declares bypass_host_loop: none. It runs per host and therefore respects serial, throttle and forks. A module that bypassed the host loop would reboot the entire play’s hosts regardless of batching, which is precisely the incident this part opens with.

What happens if the new kernel boots on the canary and panics on batch three?

You find out on batch three, which is the point, and what happens next depends entirely on decisions made before the run.

The immediate behaviour: the hosts in batch three do not come back. The reboot task fails after reboot_timeout — twice that value in the worst case, because the timeout is evaluated separately for reboot verification and for test_command success. With max_fail_percentage: 0 the play stops and batch four never starts. The always: block runs for the batch, which is what records those hosts as out of service rather than leaving them silently drained.

Then the hard part, which Ansible cannot help with: those hosts are down and unreachable. Recovery is console access — iDRAC, iLO, the hypervisor console — selecting the previous kernel from the boot menu, and bringing them back one at a time.

The reason the canary passed and batch three did not is almost always hardware or driver divergence. A different NIC firmware revision, a storage controller in one rack, a host with a GPU. Fleets that look homogeneous in inventory are frequently not homogeneous in hardware, and a kernel is the change most sensitive to that difference.

Two things follow for the design of the run. Order the batches so the canary phase spans the hardware variety rather than taking the first five hosts alphabetically, which on most naming schemes means five hosts in the same rack. And record the hardware profile alongside the kernel result, so the failure produces “6.6.9 panics on the Dell R650s with the older BIOS” rather than “6.6.9 broke some hosts”.

The one thing not to do is re-run the play hoping for a different result. The kernel that panicked will panic again, and the next batch is another six hosts on the crash cart.

Knowledge check

Why this matters

This lesson is where every control in the part is used at once, and the reason they are all needed is that a kernel upgrade removes the two things that make other changes survivable.

You cannot test it in place. A configuration change can be validated before the service reloads; a kernel cannot be validated before it is running, and it can only be running after a reboot.

You cannot undo it cheaply. A bad package can be downgraded in seconds. A bad kernel needs another reboot, on a host that may not be reachable to be told to reboot.

So the entire safety argument moves out of the change and into the process around it: a small first batch, because that is the only place the risk can be contained; a validation step that proves the new kernel is running, because “the host answered” cannot tell the difference; a stop-the-run failure policy, because there is no partial undo; and console access in the runbook, because the recovery path for the worst case does not go through Ansible at all.

Which is the honest summary of this part: Ansible executes the patch run well. It does not make the patch safe. The safety is in the shape, and the shape is a decision made before the play was written.