Proving a kernel campaign worked - CVEs, backports and vulnerability state · Linux

The campaign is finished. Forty nodes have been drained, patched, rebooted, validated and returned. Somebody now has to answer a question from a security or compliance function: is the fleet patched for CVE-2025-XXXXX?

That question is harder to answer well than the campaign was to run, and the two most obvious answers are both wrong. “The package is installed” is wrong because installed is not running. “The kernel is version 5.14.0” is wrong for a reason that catches out almost everyone the first time.

Version strings do not mean what the scanner thinks

Enterprise distributions do not ship the upstream kernel that the CVE was fixed in. They take a kernel from years ago, keep its version number for the lifetime of the release, and backport individual fixes into it.

A Red Hat Enterprise Linux 9 host reports 5.14.0-427.el9. The upstream fix for a given CVE may have landed in 6.6. Both facts are true simultaneously, because Red Hat backported the patch into their 5.14.0 tree and expressed that in the package release field — the -427 — not in the version.

The consequence is that a scanner reasoning “this host runs 5.14, the fix is in 6.6, therefore vulnerable” flags every RHEL 9 host in the estate, permanently, and will continue to do so after you upgrade because the version never changes. The same applies to Ubuntu LTS, SLES and Debian stable.

Arguing about the scanner output is the wrong fight

The instinct is to explain backporting to whoever runs the scanner and have the finding dismissed. That works once and recurs every month.

The durable answer is to produce evidence in a form the scanner cannot contradict: the vendor advisory identifier that covers the CVE, and the package release your hosts are running, which is greater than or equal to the one the advisory names. RHSA, USN and DSA identifiers are the vendor’s own statement that a given build contains a given fix. That is a stronger claim than a version comparison, and it is auditable.

Configure the scanner to use its vendor-aware data feed if it has one — most do — and treat a scanner that only does version comparison against upstream as a tool that cannot assess Linux hosts, because that is what it is.

Mapping a CVE to a build

The changelog is the authoritative local record. It is on every host, it does not need a network, and it names the CVE explicitly.

Read-only / SafeCVE to build mapping— Both commands read local metadata and change nothing. The RHEL form queries the changelog of the exact running kernel, which is the one that matters - querying kernel without a version returns the newest installed, which may not be running. The Debian changelog lives with the package documentation and is compressed, hence zgrep. Note that a changelog entry proves the fix is in that build; its absence proves only that this build does not mention it, which is not the same as vulnerable.

$ # RHEL family - the running kernel's own changelog
rpm -q --changelog "kernel-$(uname -r)" | grep -i 'CVE-2025-' | head

# Debian family - the installed kernel package changelog
dpkg-query -W -f='${binary:Package}\n' 'linux-image-*' | grep "$(uname -r)"
zgrep -i 'CVE-2025-' "/usr/share/doc/linux-image-$(uname -r)/changelog.Debian.gz" | head

- redhat: fix CVE-2025-XXXXX (Foo Bar) [RHEL-98765] {CVE-2025-XXXXX}
- net: sched: fix CVE-2025-YYYYY (Baz Qux) [RHEL-98766] {CVE-2025-YYYYY}

linux-image-6.8.0-51-generic
* CVE-2025-XXXXX
  - net/sched: reject invalid parent handle

Illustrative output

Note the deliberate detail in the RHEL command: kernel-$(uname -r), not kernel. Querying the package name alone returns the newest installed kernel, which on a host awaiting its reboot is precisely the kernel that is not running — the same installed-versus-running confusion the rollback lesson opens with.

On Ubuntu with a Pro subscription there is a direct tool that resolves the CVE against the vendor data and reports the state of this host:

sudo pro fix CVE-2025-XXXXX

It reports whether the host is affected, whether a fix is available, and whether it is already applied — including fixes delivered by livepatch rather than by a reboot, which is the case the next section is about.

What `/sys/devices/system/cpu/vulnerabilities` tells you

For the CPU-side speculative execution issues, the kernel publishes its own assessment at runtime. This is the strongest form of evidence available, because it is not a version comparison at all — it is the running kernel stating what it has detected and what it has enabled.

Read-only / Safekernel vulnerability state— Read-only sysfs. Three answers are possible per entry. Not affected means this CPU is not susceptible - the commonest answer, and it depends on the processor model, not on the kernel version. Mitigation: followed by a description means the kernel detected the issue and enabled a defence. Vulnerable means the kernel knows about the issue and is not defending against it, which is the only one that needs action. The set of files present grows with each kernel release, so a host missing an entry another host has is running an older kernel that does not know about that issue.

$ grep -r . /sys/devices/system/cpu/vulnerabilities/

/sys/devices/system/cpu/vulnerabilities/meltdown:Not affected
/sys/devices/system/cpu/vulnerabilities/mds:Not affected
/sys/devices/system/cpu/vulnerabilities/spectre_v1:Mitigation: usercopy/swapgs barriers and __user pointer sanitization
/sys/devices/system/cpu/vulnerabilities/spectre_v2:Mitigation: Enhanced / Automatic IBRS; IBPB: conditional; STIBP: disabled
/sys/devices/system/cpu/vulnerabilities/spec_store_bypass:Mitigation: Speculative Store Bypass disabled via prctl
/sys/devices/system/cpu/vulnerabilities/retbleed:Not affected
/sys/devices/system/cpu/vulnerabilities/srbds:Not affected
/sys/devices/system/cpu/vulnerabilities/old_microcode:Not affected

Illustrative output

Two things this output is often over-read for.

It covers hardware vulnerabilities only. Speculative execution, microarchitectural data sampling, and related CPU issues. A use-after-free in the network stack does not appear here and never will. This file set is a strong answer to one class of question and silent on the rest of the kernel.

A missing entry is a signal. old_microcode and the newer entries only exist on kernels that know about those issues. A fleet where some hosts have an entry and others do not is a fleet running two kernel generations — which the campaign was supposed to eliminate, and which is worth reconciling.

`mitigations=off` makes a fully patched kernel report Vulnerable

The kernel command line accepts mitigations=off, which disables the speculative-execution defences wholesale. It is sometimes set deliberately on isolated compute nodes where the performance cost is unacceptable and the threat model genuinely does not include untrusted code on the machine.

It is much more often set years ago, by somebody who has left, for a benchmark, and then baked into a golden image.

The effect is that every relevant entry reads Vulnerable regardless of how current the kernel is. No amount of patching changes it, and a campaign completed perfectly produces a fleet that reports vulnerable to any tool reading these files.

Check it explicitly as part of the campaign close:

grep -o 'mitigations=[^ ]*' /proc/cmdline
grep -l Vulnerable /sys/devices/system/cpu/vulnerabilities/*

If mitigations=off is present, it is either a documented decision with a named owner and a threat-model justification, or it is a finding. There is no third category. Removing it is a kernel command line change, so it takes a reboot — fold it into the campaign rather than scheduling a second one.

Live-patched hosts break every simple check

A host that took a live kernel patch is the case that defeats both of the obvious verification methods at once:

uname -r reports the old kernel, because the host has not rebooted. A check keyed on the running version says vulnerable.
The kernel package on disk may also be the old one, because the fix arrived as a patch module rather than as a new kernel. A check keyed on installed packages also says vulnerable.

The host is, in fact, fixed. The evidence is in the live patching state, and it has to be collected separately:

# RHEL family
sudo kpatch list

# Ubuntu
sudo canonical-livepatch status --verbose

This is the strongest argument for the policy in linux-live-kernel-patching-concepts: live patching buys time against the CVE deadline, and the reboot still happens on the normal schedule. A fleet that live-patches and never reboots accumulates hosts whose true state cannot be read from any single command, and whose live patch is silently discarded the next time anything reboots them.

The campaign record

Collect this per host, at the end of the campaign, and keep it. It is what you hand to an auditor and what you check the scanner’s output against.

#!/usr/bin/env bash
# /usr/local/sbin/kernel-campaign-record
set -euo pipefail

printf 'host=%s\n'            "$(hostname -f)"
printf 'running_kernel=%s\n'  "$(uname -r)"
printf 'newest_installed=%s\n' "$(ls -1v /boot/vmlinuz-* | tail -1 | sed 's|.*/vmlinuz-||')"
printf 'boot_time=%s\n'       "$(uptime -s)"
printf 'cmdline=%s\n'         "$(cat /proc/cmdline)"

printf 'vulnerable_entries=%s\n' \
  "$(grep -l Vulnerable /sys/devices/system/cpu/vulnerabilities/* 2>/dev/null | wc -l)"

if command -v kpatch >/dev/null; then
  printf 'livepatch=%s\n' "$(kpatch list | tr '\n' ' ')"
fi

The field that closes the campaign is the comparison between running_kernel and newest_installed. Equal means the host is done. Different means the host has the patch installed and is still running the old kernel — the single most common way a campaign is reported complete while a third of the fleet is unprotected.

# Fleet-wide, from the control node
ansible -i /etc/ansible/prod all -m ansible.builtin.command \
  -a "/usr/local/sbin/kernel-campaign-record" \
  | grep -E 'running_kernel|newest_installed'

Version strings do not mean what the scanner thinks

Mapping a CVE to a build

What /sys/devices/system/cpu/vulnerabilities tells you

Live-patched hosts break every simple check

The campaign record

Knowledge check

What `/sys/devices/system/cpu/vulnerabilities` tells you