Runtime versus on-disk drift - when the file is right and the system is not

Every drift detector so far compares files: against the playbook, against the package, against a baseline, against a peer. All of them share an assumption — that the file describes the system.

For a large part of a Linux host, it does not. Firewall rules, kernel parameters, loaded modules and systemd overrides all have a loaded state and an on-disk state, and they can disagree indefinitely without any file being wrong.

This drift is quiet in a specific way: it has no effect until the host reboots, and then all of it applies at once. That is why “we rebooted a host that had been up for two years and it did not come back” is a category of incident rather than an anecdote.

Firewall: rules loaded, rules saved

nft applies rules to the running kernel immediately. Nothing writes them to disk unless you do.

# What the kernel is actually enforcing right now
sudo nft list ruleset

# What will be loaded at boot
sudo cat /etc/nftables.conf

A rule added during an incident with nft add rule ... is enforcing traffic policy and is absent from the file. It disappears at the next reboot, which is the good case — the bad case is the reverse: a rule that was removed live to unblock a service is still in the file, and comes back at the next boot, breaking the service again months later with no change to blame.

# Substitute your own values before running:
SAVED=/etc/nftables.conf

# Diff loaded against saved. Ignore counters and handles, which
# differ legitimately on every run.
diff <(sudo nft -s list ruleset | grep -vE 'counter packets|# handle') \
     <(grep -vE 'counter packets|# handle' "$SAVED")

The iptables-era equivalent, where the saved file is produced by a separate command entirely:

# Loaded rules, in save format
sudo iptables-save > /var/tmp/iptables-loaded.txt

# Compare against whatever the distribution restores at boot
diff /var/tmp/iptables-loaded.txt /etc/iptables/rules.v4

With firewalld the split is explicit and built in: every change is either runtime or permanent, and --permanent changes do not take effect until reloaded.

# Two different answers to "what are the rules"
firewall-cmd --list-all
firewall-cmd --list-all --permanent

sysctl: the value, the file, and the shadow

sysctl -w sets a kernel parameter now. Files under /etc/sysctl.d/ set it at boot. Neither knows about the other.

$ sysctl -n net.ipv4.ip_forward; grep -rs 'ip_forward' /etc/sysctl.conf /etc/sysctl.d/

1
/etc/sysctl.d/99-hardening.conf:net.ipv4.ip_forward = 0

That output is a host that will change behaviour at its next reboot, and the change is a routing change. Nothing in /etc is wrong; the file is exactly what the playbook says it should be, so a CM dry-run reports a clean host.

There is a second, subtler form. Files in /etc/sysctl.d/ are applied in lexical order, so a later-sorting file silently overrides an earlier one. A hardening file numbered 50- is defeated by a vendor file numbered 99-, and both files are present and correct in every diff.

# Which files exist, in the order they are applied
ls -1 /etc/sysctl.d/ /usr/lib/sysctl.d/ 2>/dev/null | sort

# Substitute your own values before running:
KEY=net.ipv4.conf.all.rp_filter

# Every declaration of one key, across every source
grep -rns -- "$KEY" /etc/sysctl.conf /etc/sysctl.d/ /usr/lib/sysctl.d/ 2>/dev/null
sysctl -n "$KEY"

Reconciling is one command, and it is the safe direction — it makes the running system match the files, rather than the reverse:

# Re-apply every file. Read the output: each line it changes is
# a runtime value that had drifted from the declared one.
sudo sysctl --system

systemd: effective now, absent after reboot

The unit file on disk is not the whole configuration. Drop-ins, runtime overrides and un-reloaded edits all produce a gap between what the file says and what systemd is enforcing.

# Substitute your own values before running:
UNIT=ssh.service

# The effective configuration, assembled from every source
systemctl show -p FragmentPath -p DropInPaths "$UNIT"

# The merged unit as systemd sees it
systemctl cat "$UNIT"

Three specific gaps to look for:

1. Runtime drop-ins under /run/systemd/system. These are created by systemctl set-property --runtime and by systemctl edit --runtime, they are effective immediately, and /run is tmpfs so they vanish at reboot. A resource limit applied this way during an incident is load-bearing configuration that exists nowhere on disk.
2. Edited unit files with no daemon-reload. The file on disk is the new version; the running systemd still holds the old one. systemctl status warns about this, and the warning is easy to miss during an incident.
3. Masked or disabled units. A unit masked to stop it interfering during an incident stays masked, and systemctl status for it reports the mask rather than a failure - so a monitoring agent masked six months ago looks like a deliberate configuration.

# Runtime-only overrides: effective now, gone at reboot
sudo find /run/systemd/system -type f 2>/dev/null

# Units whose on-disk file has changed since systemd loaded it
systemctl list-units --state=running | head -20
sudo systemctl daemon-reload   # after reviewing, not before

# Anything masked
systemctl list-unit-files --state=masked

The systemd-delta command from linux-how-drift-accumulates belongs here too: it names every shipped unit that a local file overrides or extends, which is the on-disk half of the same question.

Modules and kernel command line

Two more places where the running state has its own source:

# Loaded now
lsmod | head

# Configured to load at boot
grep -rhv '^\s*#' /etc/modules-load.d/ /etc/modules 2>/dev/null | grep -v '^$'

# Explicitly blocked from loading
grep -rhs '^blacklist\|^install' /etc/modprobe.d/ 2>/dev/null

# The kernel command line in force right now
cat /proc/cmdline

# What the bootloader will use next time
grep -s 'GRUB_CMDLINE_LINUX' /etc/default/grub

A module loaded by hand during troubleshooting, or a kernel parameter set once with a grubby/kexec variant and never written to /etc/default/grub, is the same pattern: correct now, absent after the reboot.

Knowledge check