Break/Fix: one server never came back from a kernel patch and the pre-flight check said it had plenty of room · Break/Fix · Ansible

Reported symptoms

The monthly kernel patch runs across 40 application hosts at 01:00. It completes on 39. The fortieth times out.

TASK [Reboot into the new kernel] **********************************************
fatal: [app017]: FAILED! => {"changed": false, "msg": "Timed out waiting for last boot time check (timeout=600)"}

Twenty minutes later app017 still does not answer SSH or ICMP. The on-call engineer power-cycles it through the management processor. It comes up and stops in the same place.

What makes this confusing rather than routine:

The play has a pre-flight check that asserts free space on /boot. It passed on this host.
The package upgrade task on this host reported success.
The other 39 hosts are running the new kernel and are perfectly healthy.

So the first theory is hardware, and twenty minutes go into the management processor logs, which are clean.

Evidence provided

Read-only / Safethe guard passed and the upgrade succeeded

$ grep -B2 -A6 'app017' logs/patch-0100.log | head -20

TASK [Assert enough free space on /boot] ***************************************
ok: [app017]

TASK [Install the new kernel] **************************************************
changed: [app017]

TASK [Reboot into the new kernel] **********************************************
fatal: [app017]: FAILED! => {"changed": false, "msg": "Timed out waiting for last boot time check (timeout=600)"}

Read-only / Safethe upgrade said so, in output nobody reads on a green task

$ grep -A4 'Install the new kernel' logs/patch-0100-verbose.log | grep -i 'warning\|space'

update-initramfs: Generating /boot/initrd.img-6.8.0-52-generic
zstd: error 70 : Write error : cannot write block : No space left on device
E: mkinitramfs failure zstd 70
update-initramfs: failed for /boot/initrd.img-6.8.0-52-generic with 1.

Read-only / Safethe guard reads a fact

$ grep -n -A6 'Assert enough free space' patch.yml

8:- name: Assert enough free space on /boot
9:  ansible.builtin.assert:
10:    that:
11:      - (ansible_mounts | selectattr('mount','equalto','/boot') | first).size_available > 200000000
12:    fail_msg: "Not enough free space on /boot to install a kernel"

Read-only / Safefacts are cached for a week

$ ansible-config dump | grep -i 'fact_caching'

CACHE_PLUGIN(/srv/automation/ansible.cfg) = jsonfile
CACHE_PLUGIN_CONNECTION(/srv/automation/ansible.cfg) = /var/cache/ansible/facts
CACHE_PLUGIN_TIMEOUT(/srv/automation/ansible.cfg) = 604800

Read-only / Safewritten three days ago, describing 400 MB of free space

$ stat -c '%y' /var/cache/ansible/facts/app017; jq '.ansible_mounts[] | select(.mount=="/boot") | .size_available' /var/cache/ansible/facts/app017

2026-08-08 03:11:44.000 +0000
418906112

Read-only / Safea comparable host, measured now - the cached figure was never true this week

$ ansible app018 -i inventory -b -m ansible.builtin.command -a 'df -h /boot'

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2       512M  474M   11M  98% /boot

Work the evidence before reading on

The guard passed and the host is unbootable. Both of those are facts, so the guard was answering a question about something other than the current state of the host.

Compare the free-space figure the guard used against the figure a comparable host reports today. What is the difference, and what happened in between?
The upgrade task reported changed and success. Read the verbose output of that same task. Do the two agree?
The other 39 hosts returned. Does that tell you the guard worked on them, or only that they had more room?

Before continuing: when was the value the guard asserted on actually measured, and by what?

Root cause

1. The guard asserted on a three-day-old measurement

The assertion reads ansible_mounts, which is a fact. Facts on this controller are cached, with a timeout of a week.

The cached entry for app017 was written three days earlier. At that time /boot had roughly 400 MB free. Since then the previous month’s kernel had been installed and had consumed most of it, so the real figure on the night of the run was around 11 MB.

The guard compared 400 MB against its threshold, passed, and reported ok. Everything about that is correct behaviour: the fact was in the cache, the cache was inside its timeout, and the assertion evaluated a true statement about a filesystem that had not existed for three days.

A cached fact is a claim about a host in the past. Using one for a safety decision converts the guard into a record of history.

2. The upgrade succeeded and the initramfs did not

The package manager installed the kernel package. Its post-installation step, which builds the initial ramdisk, ran out of space and failed.

Whether that failure propagates to a non-zero exit status depends on the distribution and the specific trigger, and in this case it did not - the package was configured, the upgrade reported success, and the module reported changed. The failure text was in the module output, which nobody reads on a task that reports success.

The host was left with a boot loader entry pointing at a new kernel and a truncated ramdisk.

3. The reboot task did its job and could not have helped

ansible.builtin.reboot issues the reboot, waits for the host to respond, and runs a test command to confirm it is usable. The default timeout is 600 seconds and the default test command is whoami.

None of that can detect a host that will never come back. The module waited the full timeout and reported it accurately. The failure message is correct and uninformative, because from the controller’s point of view a host that stops in its initramfs and a host whose power supply failed look identical.

Under the hood

The reboot module is more subtle than it appears, and two of its defaults are worth knowing before relying on it for kernel work.

boot_time_command defaults to reading the kernel boot identifier, and the module compares that value before and after. That is why the failure message mentions a boot time check: the module is not merely waiting for SSH, it is waiting to see evidence that the machine actually rebooted rather than that the connection came back.

test_command defaults to whoami. It runs after the host answers, and it establishes that a login shell works. It does not establish that the machine booted the kernel you intended, that services started, or that anything else about the host is correct.

That distinction produces the quieter sibling of this incident. A host whose boot loader default was not updated will reboot promptly, answer SSH, pass whoami, and be reported as a completely successful patch - while running the old kernel. Nothing in the play notices, and the host stays unpatched through every subsequent run that checks only whether the package is installed.

The check that catches both is the same one: after the reboot, compare the running kernel version against the installed one. It is one task and it converts reboot from an action into a verified step.

A guard that reads a cached fact is worse than no guard

Without the assertion, somebody might have checked /boot by hand before a fleet kernel patch. With it, nobody did, because the play appeared to be checking.

Fact caching is a legitimate and valuable optimisation - gathering facts across a large fleet is often the single largest cost in a run. The danger is narrow and specific: a cached fact is a measurement from the past, and safety decisions need a measurement from now.

The distinction is easy to apply. Facts that describe what a host is - architecture, distribution family, network interface names - change rarely and cache well. Facts that describe what a host has right now - free space, free memory, uptime, mounted filesystems, running services - are exactly the ones a guard cares about and exactly the ones that go stale.

If a play makes a decision on a measured value, measure it in the play. A command running df with a registered result is three lines and cannot be stale.

Resolution

Connect to the console or management processor. There is no network path to a host stopped in its initramfs, and every step below needs one.
Confirm the diagnosis from the console. An initramfs prompt with a message about being unable to find the root device is this fault; a kernel panic or a failed hardware check is a different incident with a different fix.
Boot the previous kernel from the boot loader menu to get a working system. Do not try to repair anything from the initramfs prompt if a working kernel is available.
Reclaim space on /boot by removing superseded kernels through the package manager, so the boot loader configuration is regenerated correctly. Deleting files by hand leaves the boot loader referring to entries that no longer exist.
Regenerate the initramfs for the new kernel and confirm it completes without warnings and produces a file comparable in size to the one for the previous kernel.
Reboot once more and confirm the host returns unattended into the new kernel. Watching it come back is the only proof that matters.
Fix the guard: measure free space during the run rather than reading a fact, and place the assertion immediately before the upgrade rather than at the top of the play.
Fix the upgrade step so a failed initramfs generation fails the task, and add a post-upgrade check that the initramfs exists and is plausible before any reboot is attempted.

Verification

The guard can fail. On a scratch host, fill /boot deliberately and confirm the pre-flight refuses to start the upgrade. This is the check that would have prevented the incident and it had never been exercised.
The recovered host reboots unattended. Reboot it once more and confirm it returns without console intervention; a host that needed a person to come back is not recovered.
The running kernel matches the installed kernel. Compare the running version against the newest installed version on every host in the fleet, not just the repaired one. This catches the silent variant where a host reboots successfully into the old kernel.
The initramfs is intact. The file for the running kernel exists and is comparable in size to the one for the previous kernel; a truncated ramdisk is often the only visible difference.
Free space is adequate, measured now. Run a df across the fleet in this run rather than reading any cached figure, and require headroom for at least one more kernel.
A failed initramfs generation fails the task. Reproduce it on a scratch host and confirm the play stops before the reboot rather than after it.
Out-of-band access works on every host that will be rebooted. Test it before the next fleet patch, not during it.

Prevention

Guards measure, they do not remember. Any assertion about a resource that can change needs a value gathered in this run:

- name: Measure free space on /boot now
  ansible.builtin.command: df --output=avail --block-size=1 /boot
  changed_when: false
  register: boot_free

- name: Refuse to install a kernel without room for its initramfs
  ansible.builtin.assert:
    that: boot_free.stdout_lines[1] | int > 300000000
    fail_msg: "Only {{ boot_free.stdout_lines[1] | int }} bytes free on /boot; refusing to install a kernel."

Put the guard immediately before the dangerous step. A check at the top of a play is a check about a state that several tasks have had a chance to change.
Refresh or bypass the fact cache for plays that make safety decisions. Caching is worth having; caching a value a guard depends on is not.
Treat a reboot as irreversible and give it an explicit precondition: the kernel installed, the initramfs generated and verified, the boot loader updated, and free space still adequate.
Verify the running kernel after every patch reboot. It is the check that catches both the host that never returns and the host that returns into the wrong kernel.
Size /boot for at least three kernels and prune superseded ones as part of the patch cycle. Almost every incident of this shape is a 512 MB /boot that was adequate two kernel sizes ago.
Ensure out-of-band access exists and is tested before any fleet reboot. One host in forty not returning is the expected case, not the surprising one.

What if the host had returned promptly but running the old kernel?

Far more common, entirely silent, and arguably worse.

If /boot had been full enough to prevent the boot loader configuration being updated, but the machine still had a valid default entry, the host would have rebooted in ninety seconds, answered SSH, passed the reboot module test command, and been reported as a successful patch. It would be running the old kernel with the old vulnerabilities, and the package database would say the new kernel is installed.

Every subsequent check that asks “is the patched package installed” would answer yes. Compliance reporting would show the host as patched. The only thing that disagrees is the running kernel version, and nothing was looking at it.

That is why the post-reboot check compares running against installed rather than merely confirming the host came back. The comparison costs one task and it is the difference between a patch process and a patch process you can attest to.

A third variant is worth mentioning because it is the one that catches people out on cloud instances: the host returns, runs the new kernel, and a module the workload depends on was not rebuilt for it. The machine is up, the kernel is right, and a service fails to start. The lesson is the same - a reboot that completes is not a reboot that succeeded, and the play has to say what success means.

Why this matters

The guard was there. Somebody had thought about this failure, written an assertion to prevent it, and put it at the top of the play. It passed, and the failure happened anyway, because the value it asserted on had been measured three days earlier by a different run for a different purpose.

That is the point worth carrying. Fact caching is an optimisation that changes the meaning of every fact from “what is true” to “what was true when we last looked”, and the facts most worth caching are the stable ones while the facts guards care about are the volatile ones. A safety check built on a cached measurement produces exactly the confidence that prevents anybody from checking by hand.

The second point is about what success means. The package manager said success, the module said changed, the play said green - and the host was unbootable. Between “the tool reported success” and “the outcome you wanted happened” there is always a gap, and on an irreversible step like a reboot, that gap has to be closed by a check that runs afterwards and compares reality against intent.

Reported symptoms

Evidence

Root cause

Remediation

Verification

Prevention

Reported symptoms

Evidence provided

Work the evidence before reading on

Root cause

1. The guard asserted on a three-day-old measurement

2. The upgrade succeeded and the initramfs did not

3. The reboot task did its job and could not have helped

Resolution

Verification

Prevention