Reboot and restart requirements - finishing the patch

apt upgrade completed without errors. The new libssl3 is on disk. The scanner will report this host as patched at its next run.

Every process that was running before the upgrade is still executing the old library. The kernel does not swap a mapped file out from underneath a running process; it keeps the old inode alive, unlinked from the directory tree, until the last reference closes. Until each of those processes restarts, the vulnerability is exactly as exploitable as it was an hour ago.

Installing the package is the middle of the job.

Three classes of update

The first row is the one that gets missed, because nothing fails and nothing logs. A web server that mapped the old OpenSSL keeps serving traffic perfectly well with the vulnerable code in its address space.

Finding deleted-but-mapped files

When a package replaces a library, the old inode stays alive for anything that had it open. lsof marks those mappings DEL:

$ lsof -d DEL 2>/dev/null | head -4

COMMAND       PID    USER  FD   TYPE DEVICE SIZE/OFF    NODE NAME
dbus-daem 1349147 ebrandi DEL    REG    8,2          1083747 /usr/lib/x86_64-linux-gnu/libnss_systemd.so.2
dbus-daem 1349147 ebrandi DEL    REG    8,2          1083821 /usr/lib/x86_64-linux-gnu/libsystemd.so.0.42.0

Run it as root to see the whole host, and reduce it to the set of units you have to act on:

# Every process holding a deleted mapping, one PID per line
sudo lsof -d DEL 2>/dev/null | awk 'NR > 1 { print $2 }' | sort -un

# Which systemd units those PIDs belong to
sudo lsof -d DEL 2>/dev/null | awk 'NR > 1 { print $2 }' | sort -un |
    while read -r pid; do
        systemctl status "$pid" --no-pager 2>/dev/null | head -1
    done

systemctl status <pid> resolving a PID to its unit is the useful half of that: it turns “these 40 processes” into “these 6 services”.

needrestart

needrestart does the same analysis properly, including interpreters whose loaded scripts changed, and it knows about kernels and microcode. Batch mode makes it scriptable:

# needrestart -b -k

NEEDRESTART-VER: 3.11
NEEDRESTART-KCUR: 7.0.0-29-generic
NEEDRESTART-KEXP: 7.0.0-29-generic
NEEDRESTART-KSTA: 2

The kernel status codes are the answer:

Anything other than 1 means the running kernel is not the installed kernel. Read KSTA, not KCUR against KEXP: comparing the two release strings is a string comparison that does not capture every case the tool is checking.

The other batch keys:

# Services holding obsolete libraries
sudo needrestart -b -l | grep '^NEEDRESTART-SVC:'

# Logged-in sessions running obsolete binaries - restarting
# these logs somebody out, so they are reported separately
sudo needrestart -b | grep '^NEEDRESTART-SESS:'

# Containers, reported but never restarted by needrestart
sudo needrestart -b | grep '^NEEDRESTART-CONT:'

# CPU microcode
sudo needrestart -b -w

Batch mode never restarts anything and never opens a dialog, which makes it safe to run from monitoring. -r l is the list-only restart mode for interactive use.

Determining whether a reboot is pending

Debian family. The package scripts create a flag file, and a second file naming the packages responsible:

test -f /var/run/reboot-required && echo "reboot pending"
cat /var/run/reboot-required.pkgs 2>/dev/null

.pkgs matters for the decision: a reboot pending because of linux-image is a security matter, one pending because of a firmware package for hardware you do not use is not.

RHEL family. The plugin answers by exit code, which makes it directly usable in a script:

# Exit 0: no reboot required. Exit 1: reboot required.
if ! dnf needs-restarting -r; then
    echo "reboot pending"
fi

# Services that need restarting, without rebooting
dnf needs-restarting -s

Either family. The kernel question, independent of any helper:

uname -r
dpkg -l 'linux-image-*' 2>/dev/null | awk '/^ii/ { print $2, $3 }'
rpm -q kernel 2>/dev/null

Useful as a cross-check, and not sufficient on its own - needrestart -b -k and needs-restarting -r look at more than the release string.

Live patching

Live patching applies a fix to the running kernel without a reboot: Canonical Livepatch on Ubuntu, kpatch on the RHEL family, Ksplice on Oracle Linux.

What it does: patches kernel function text for a defined set of CVEs, chosen by the vendor.

What it does not do:

• Cover every kernel CVE. Fixes that change data structure layouts generally cannot be live patched.
• Update the kernel on disk. The next boot still uses whatever is installed, so live patching defers a reboot rather than removing one.
• Cover userspace at all. A live-patched kernel does nothing for the OpenSSL your web server has mapped.
• Cover CPU microcode, which needs a reboot to load reliably.

Treat it as what it is: a way to move a reboot out of an emergency window and into a planned one. A host that has been live patched for eight months has accumulated eight months of divergence between the running kernel and the installed one, and that divergence is itself a risk at the next boot.

Deciding what to restart

Restarting everything that appears in the list is not always right, and the exceptions are worth knowing.

• sshd is safe to restart. On Debian and Ubuntu the unit sets KillMode=process, so restarting it replaces the listener and leaves existing sessions alive. Verify before you rely on it:

systemctl show ssh.service -p KillMode

• systemd itself needs a re-exec, not a restart.

sudo systemctl daemon-reexec

• dbus and systemd-logind take sessions with them. Restarting them on a host with interactive users disconnects them, and on some systems destabilises anything holding a bus connection. These are usually a reason to reboot rather than restart.
• Databases and stateful services get restarted on their own terms, with the drain and failover procedure that applies to them, not because a library changed.
• Containers appear in NEEDRESTART-CONT and are not restarted by the host tooling. A container carries its own copy of the library; patching the host does nothing for it. That is an image rebuild, not a restart.

The needrestart blocklists exist for exactly these cases and live in /etc/needrestart/conf.d/:

grep -rn 'blacklist' /etc/needrestart/needrestart.conf | head
ls /etc/needrestart/conf.d/

Pending reboot is a backlog, not an event

The final failure is organisational. A pending reboot is a state that persists, and states with no owner persist forever.

Make it measurable:

• Export reboot pending, yes or no and uptime per host.
• Alert on “reboot pending for more than N days”, where N comes from the environment tier, not from convenience.
• Report the count of hosts with obsolete libraries loaded, alongside the count of unpatched packages. The two numbers should converge after each patch window; if they do not, the restart step is being skipped.
• Give the reboot a scheduled home. The rolling-maintenance process in the cluster part is where quorum-aware reboots belong.

Knowledge check