Deciding whether a reboot is required · Ansible

Rebooting every host you patched is wasteful. Rebooting no host you patched is dangerous. The interesting work is in between, and it starts with a question the host can answer for itself: do you need a reboot?

The two distribution families answer it by entirely different mechanisms, and neither one is a superset of the other. Writing a play that treats one as universal produces a fleet where half the hosts are never rebooted and the run log says everything is fine.

Debian and Ubuntu: a file, if something wrote it

On Debian-family hosts, a package that needs a reboot creates /var/run/reboot-required. A companion file, /var/run/reboot-required.pkgs, lists the packages that asked for it, one per line.

Both live under /run on modern systems — /var/run is a symlink — so they are tmpfs files that vanish on boot. That is the property that makes them correct: the flag cannot survive the reboot it was asking for.

Read-only / Safeasking a Debian-family host whether it needs a reboot— Read-only. stat inspects the filesystem and changes nothing. slurp reads a file's contents back to the controller; it writes nothing on the host.

- name: Check for the Debian-family reboot flag
ansible.builtin.stat:
  path: /var/run/reboot-required
  get_checksum: false
register: reboot_flag
when: ansible_facts.os_family == 'Debian'

- name: Read which packages requested the reboot
ansible.builtin.slurp:
  src: /var/run/reboot-required.pkgs
register: reboot_pkgs_raw
when:
  - ansible_facts.os_family == 'Debian'
  - reboot_flag.stat.exists | default(false)
failed_when: false

- name: Record the reason
ansible.builtin.set_fact:
  reboot_required: '{{ reboot_flag.stat.exists | default(false) }}'
  reboot_reason: >-
    {{ (reboot_pkgs_raw.content | default('') | b64decode).split('\n')
       | reject('equalto', '') | list }}
when: ansible_facts.os_family == 'Debian'

get_checksum: false is worth setting. stat defaults it to true, and checksumming a file you only want the existence of is pure cost — trivial here, and not trivial when the same habit is applied to a large file later.

The flag only exists if something is there to write it

/var/run/reboot-required is not a kernel feature. It is created by a shell script, notify-reboot-required, shipped in the update-notifier-common package and invoked from package maintainer scripts.

On a host where update-notifier-common is not installed — minimal Debian images, container-derived base images, anything built to be small — the file is never created, no matter what is patched. Your detection returns “no reboot needed” on a host that just took a new kernel.

This is a false negative, and it is the worst shape of bug in this lesson: it is silent, it is per-host, and it produces a fleet that looks correctly managed while accumulating hosts running a kernel that has not been the installed kernel for months.

Two defences. Assert the mechanism exists rather than assuming it, and never rely on the flag alone for kernels — the running-versus- installed comparison later in this lesson does not depend on any package being installed.

Read-only / Safeproving the detection mechanism is present before trusting it— Read-only. Fails the host if the tool that creates the reboot flag is absent, converting a silent false negative into a loud precondition failure.

- name: Confirm the reboot-required mechanism is installed
ansible.builtin.stat:
  path: /usr/share/update-notifier/notify-reboot-required
  get_checksum: false
register: notify_helper
when: ansible_facts.os_family == 'Debian'

- name: Refuse to trust reboot detection on a host that cannot signal it
ansible.builtin.fail:
  msg: >-
    {{ inventory_hostname }} has no notify-reboot-required helper, so
    /var/run/reboot-required will never appear regardless of what is
    patched. Install update-notifier-common or exclude this host from
    flag-based detection.
when:
  - ansible_facts.os_family == 'Debian'
  - not (notify_helper.stat.exists | default(false))

Debian-family hosts may additionally have needrestart, which answers a different and complementary question: which running daemons are using libraries that have been replaced. Its -b option enables batch mode, documented as “don’t restart anything and produce machine-readable output”, and -r l sets list-only restart mode. That tells you about service restarts, not about kernel reboots, and the two decisions are genuinely separate.

RHEL, Rocky and Alma: an exit status, and it is inverted

The RHEL-family mechanism is needs-restarting, from dnf-plugins-core. With -r it answers only the reboot question, and it answers by exit code.

The documentation is explicit, and the convention is backwards from everything else in a playbook:

Only report whether a reboot is required (exit code 1) or not (exit code 0).

Exit 1 means a reboot is needed. Exit 0 means it is not. To Ansible, a non-zero return code is a task failure, so a naive task fails on precisely the hosts you were trying to identify.

Read-only / Safehandling the inverted exit status— Read-only. needs-restarting -r inspects the RPM database and running processes; it restarts nothing. failed_when: false stops Ansible from treating the informative exit code as a failure.

- name: Ask whether a reboot is required
ansible.builtin.command:
  argv: [dnf, needs-restarting, '-r']
register: needs_reboot
changed_when: false
failed_when: false
when: ansible_facts.os_family == 'RedHat'

- name: Record the decision
ansible.builtin.set_fact:
  reboot_required: '{{ needs_reboot.rc == 1 }}'
when: ansible_facts.os_family == 'RedHat'

Three deliberate choices in that task.

changed_when: false because asking a question is not a change. A command task reports changed by default, and an inspection task that colours the run yellow trains people to ignore yellow.

failed_when: false because the exit code is data, not a verdict. Without it, exit 1 fails the host and — under max_fail_percentage: 0 — stops the play, on the hosts that most needed patching.

argv: rather than a command string because the argument list is explicit and nothing goes near a shell. The command module does not invoke a shell anyway, and argv makes that visible.

needs-restarting is a plugin, and dnf5 does not list it

needs-restarting ships in dnf-plugins-core. On a minimal RHEL-family host that package may not be installed, and the task then fails with “No such command” rather than answering the question — a different failure from the Debian-family false negative, and a better one, because it is loud.

More significantly: the dnf-5 command index does not list a needs-restarting command. On hosts that have moved to dnf-5, verify what your distribution actually ships before building a patch programme on it. failed_when: false means such a host silently records reboot_required: false, which puts you back in false-negative territory.

The robust form distinguishes the three outcomes rather than two:

- name: Classify the reboot answer
  ansible.builtin.set_fact:
    reboot_required: '{{ needs_reboot.rc == 1 }}'
    reboot_detection_worked: '{{ needs_reboot.rc in [0, 1] }}'

Any other return code means the tool did not answer, and a host whose detection did not answer should be treated as unknown, not as clean.

needs-restarting -s (or --services) lists only the affected systemd services, which is the RHEL-family counterpart of what needrestart does on Debian: the service-restart question, distinct from the reboot question.

The comparison that works on both families

The running kernel and the installed kernel are two different things, and comparing them requires no distribution-specific tooling at all.

ansible_facts.kernel is the running kernel — verified against ansible-core 2.21.3, where a setup run reported ansible_kernel: 7.0.0-29-generic, the same string uname -r prints. It is gathered at the start of the play, which means it is the kernel that was running when facts were gathered, and after a reboot you must gather again to see the new one.

ansible.builtin.package_facts gives you what is installed. Its structure is the thing to get right: ansible_facts.packages is a dictionary keyed by package name, and each value is a list, because a host can legitimately have several versions of the same package — which is exactly the situation kernels are always in.

Read-only / Saferunning kernel versus newest installed kernel— Read-only. package_facts enumerates the package database and sets facts; it installs and removes nothing. Works on both families because both report a kernel package.

- name: Collect the installed package inventory
ansible.builtin.package_facts:
  manager: auto

- name: Work out the newest installed kernel
ansible.builtin.set_fact:
  kernel_package: >-
    {{ 'linux-image-generic' if ansible_facts.os_family == 'Debian'
       else 'kernel' }}

- name: Compare the running kernel against what is installed
ansible.builtin.debug:
  msg: >-
    running={{ ansible_facts.kernel }}
    installed={{ ansible_facts.packages[kernel_package]
                 | default([]) | map(attribute='version') | list }}

Kernel package naming is the part that is not portable

The comparison is portable. The package name is not, and this is the same lesson the package module taught in a different costume.

On RHEL-family hosts the package is kernel, and multiple versions being installed simultaneously is normal and expected. On Debian-family hosts each kernel is its own package — linux-image-6.8.0-51-generic, linux-image-6.8.0-52-generic — with linux-image-generic as a metapackage that depends on the current one.

So ansible_facts.packages['kernel'] is a list of installed kernel versions on RHEL and a KeyError on Ubuntu, while ansible_facts.packages['linux-image-generic'] is a single-element list whose version tells you which kernel the metapackage currently points at, not which kernels are on disk.

The honest way to write this is per-family, in a vars file, exactly as the previous lesson argued. The tempting way is a shell: pipeline over uname -r and rpm -q, and it is tempting precisely because it papers over a difference that is real.

For the specific question “is the running kernel the newest installed one”, ansible_facts.kernel compared against the sorted installed versions answers it directly, and it answers on a minimal host with no update-notifier-common and no dnf-plugins-core. That independence is what makes it the backstop.

Recording the decision

A reboot decision that exists only as a when: condition is invisible afterwards. Six weeks later, asked why db04 was not rebooted in the March window, nobody can answer.

Read-only / Safemaking the decision legible in the run log— Read-only. Emits one line per host stating the decision and its basis, which is what a change record and a post-incident reconstruction both need.

- name: State the reboot decision and its basis
ansible.builtin.debug:
  msg: >-
    {{ inventory_hostname }}:
    reboot_required={{ reboot_required | default('unknown') }}
    detection={{ reboot_detection_worked | default(false) | ternary('ok', 'FAILED') }}
    running_kernel={{ ansible_facts.kernel }}
    reason={{ reboot_reason | default([]) | join(', ') | default('none', true) }}

The three states that matter are true, false and unknown. Most implementations have only the first two, and they encode “the detection did not work” as false, which is the same value as “the host is fine”.

Under the hood

The two mechanisms differ in more than syntax; they answer subtly different questions, and knowing which is which prevents a class of argument about whose fleet is correctly patched.

The Debian flag is declarative and package-driven. A maintainer script decides, at install time, that this package requires a reboot, and calls notify-reboot-required. It is a statement of intent recorded by the packager. It knows nothing about what is currently running — a host that installed a new kernel and was rebooted by someone else five minutes later still has the flag, until the reboot clears it by virtue of /run being tmpfs.

needs-restarting -r is inferential and process-driven. It compares what the package database says is installed against what the running system is using. The reboot hint specifically covers the cases where a userspace restart cannot help: kernel, glibc, systemd, dbus, linux-firmware and similar core components.

The consequence is that they can legitimately disagree in both directions on equivalent hosts. Neither is wrong. They are different definitions of “needs a reboot”, and a fleet spanning both families has two definitions in play whether or not anyone has noticed.

The comparison of ansible_facts.kernel against the installed kernel packages sits underneath both: it is a direct observation of a divergence between what is on disk and what is executing, and it depends on nothing that a minimal image might have omitted. That is why it makes a good assertion and a poor sole mechanism — it catches kernels and says nothing about glibc or systemd.

What happens if you skip detection and just reboot every patched host?

It works, it is defensible, and it is what a lot of well-run fleets actually do — with one condition attached.

The argument for unconditional reboot is real: detection has false negatives, the two families disagree, and a host that reboots every month is a host whose boot path is exercised regularly rather than discovered to be broken during an incident. A machine that has been up for 400 days is a machine nobody has proven can start.

The condition is that the reboot must go through the same serial, validation and stop-the-batch discipline as everything else in this part. “Reboot every host” and “reboot every host, one batch at a time, proving each batch came back before the next starts” are not the same policy, and the first one is the incident this part opens with.

What you give up by rebooting unconditionally is time — a fleet of 300 with a serial shape that respects capacity takes considerably longer if every host reboots than if a third of them do. That is a real cost and it is a schedule cost, not a safety cost.

What you gain is the elimination of an entire failure class: no false negatives, no per-family detection differences, no host quietly running February’s kernel because a package was missing.

Both policies are defensible. What is not defensible is a policy that depends on detection while never checking that detection works.

Knowledge check

Why this matters

Conditional rebooting is an optimisation, and every optimisation trades a guarantee for efficiency.

The guarantee given up is “every patched host has been restarted into what it just installed”. What is bought is the time not spent rebooting hosts that did not need it — real, and worth having on a fleet where the window is tight.

The trade only pays if the detection is trustworthy, and this lesson is mostly a catalogue of the ways it quietly is not: a Debian flag that needs a package nobody guaranteed is installed, a RHEL plugin that may not exist on dnf-5, an exit convention that fails the task on the hosts you care about, and an “unknown” outcome that almost every implementation encodes as “no”.

So the discipline is: detect, but assert that detection worked; record the decision and its basis in the run log; and keep an independent observation — the running kernel against the installed one — that does not depend on any of it.

And if that sounds like more work than rebooting everything, that is a completely reasonable conclusion to reach. Just reach it deliberately, rather than by discovering in an audit that thirty hosts have not restarted since February.