What changed - working the highest-yield question

Most production incidents follow a change. Not all of them, but enough that asking “what moved?” before “what broke?” is the single highest-yield habit in this loop - and it is the step people skip, because it feels like administration rather than engineering.

Step 3 has one output: a time-ordered list of every change in the window that spans the first-seen time from step 1. This lesson is how to produce that list on a Linux host, and how to avoid the two ways it misleads you.

Six places a host records change

No single command answers the question. Work all six, because each one covers a class of change the others do not see.

1. Packages: what was installed, upgraded or removed, and when
2. Configuration on disk: what changed under /etc, including files a package upgrade replaced
3. systemd units: new units, changed drop-ins, and a daemon-reload that activated an edit made weeks ago
4. The running fleet automation: the last configuration management run and what it converged
5. The host itself: reboots, kernel changes, hardware and storage path changes
6. Everything off the host: deployments, DNS records, firewall rules, certificate rotations, load balancer membership, upstream provider changes

The sixth is the one that catches out engineers who only look at the host. A host that has genuinely changed nothing still breaks when the certificate it presents expires, when the DNS record it depends on moves, or when a firewall rule three racks away is tightened.

Packages

$ grep -E ' (install|upgrade|remove) ' /var/log/dpkg.log /var/log/dpkg.log.1 2>/dev/null | tail -20

/var/log/dpkg.log:2026-08-11 03:42:07 upgrade openssl:amd64 3.0.13-1 3.0.14-1
/var/log/dpkg.log:2026-08-11 03:42:11 upgrade libssl3:amd64 3.0.13-1 3.0.14-1
/var/log/dpkg.log:2026-08-11 03:42:19 upgrade nginx-common:all 1.24.0-2 1.24.0-3

On the RHEL family the same question is answered by the transaction history, which additionally tells you who ran it and gives you a transaction id you can inspect or undo.

dnf history list --reverse | tail -20
dnf history info 47
rpm -qa --last | head -20

Unattended upgrades deserve their own look, because nobody remembers them and they run in the middle of the night:

$ journalctl -u unattended-upgrades --since '24 hours ago' --no-pager | tail -15; ls -l /var/run/reboot-required 2>/dev/null

Aug 11 03:42:06 web02 unattended-upgrade[1188]: Packages that will be upgraded: libssl3 openssl nginx-common
Aug 11 03:42:31 web02 unattended-upgrade[1188]: All upgrades installed
-rw-r--r-- 1 root root 32 Aug 11 03:42 /var/run/reboot-required

Configuration on disk

With etckeeper installed, /etc is a Git repository and this question takes seconds:

git -C /etc log --since '48 hours ago' --stat
git -C /etc diff 'HEAD@{2}' -- nginx/

Without it, the filesystem timestamps are the fallback. Crude, but they answer the question far better than memory does:

$ find /etc -xdev -newermt '2026-08-10 00:00' -type f -printf '%TY-%Tm-%Td %TH:%TM %p\n' 2>/dev/null | sort | tail -20

2026-08-11 03:42 /etc/ssl/openssl.cnf
2026-08-11 03:42 /etc/nginx/nginx.conf
2026-08-11 09:05 /etc/nginx/conf.d/upstream.conf

Package upgrades that replace configuration are their own category, because the operator did not make the change and will not remember it. On Debian, .dpkg-dist and .dpkg-old files are the fingerprint; on RHEL, .rpmnew and .rpmsave.

$ find /etc -xdev -name '*.dpkg-dist' -o -name '*.dpkg-old' -o -name '*.rpmnew' -o -name '*.rpmsave' 2>/dev/null

/etc/nginx/nginx.conf.dpkg-dist
/etc/ssh/sshd_config.rpmnew

For Debian systems you can also ask which conffiles differ from what the package shipped, which is a stronger statement than a timestamp:

dpkg --verify | grep -E '^..5' | head -20

systemd units

Units change in three ways, and two of them are invisible to a find on /etc: a drop-in added under a .d directory, and a vendor unit masked or overridden.

$ systemd-delta --type=extended,overridden,masked | head -25

[EXTENDED] /usr/lib/systemd/system/nginx.service → /etc/systemd/system/nginx.service.d/override.conf
[OVERRIDDEN] /etc/systemd/system/myapp.service → /usr/lib/systemd/system/myapp.service
[MASKED] /etc/systemd/system/rsyslog.service → /dev/null
2 overridden configuration files found.

$ systemctl cat myapp.service | head -30; systemctl show myapp.service -p FragmentPath -p DropInPaths -p ExecMainStartTimestamp

# /usr/lib/systemd/system/myapp.service
[Service]
ExecStart=/usr/bin/myapp --config /etc/myapp/app.toml
# /etc/systemd/system/myapp.service.d/10-limits.conf
[Service]
MemoryMax=2G
FragmentPath=/usr/lib/systemd/system/myapp.service
DropInPaths=/etc/systemd/system/myapp.service.d/10-limits.conf
ExecMainStartTimestamp=Mon 2026-06-30 22:14:05 UTC

Configuration management

Whatever converges your fleet is a change source, and it makes changes on a schedule rather than when a human is watching.

journalctl -u ansible-pull --since '24 hours ago' --no-pager | tail -30
systemctl list-timers --all | grep -Ei 'ansible|puppet|salt|chef'

Two questions matter here, and they are different: what did the last run change, and when did the last run happen. A run that stopped happening three weeks ago is a change too - the host has been drifting since, and the symptom may be the first visible consequence.

The host

$ uptime -s; journalctl --list-boots | tail -5; uname -r; ls -lt /boot/vmlinuz-* | head -3

2026-08-11 04:03:12
-2 3f1a... Fri 2026-07-25 09:11:02 UTC—Sat 2026-08-09 02:59:41 UTC
-1 8c2d... Sat 2026-08-09 03:01:10 UTC—Tue 2026-08-11 03:59:02 UTC
0 a91b... Tue 2026-08-11 04:03:12 UTC—Tue 2026-08-11 09:20:44 UTC
6.1.0-23-amd64
-rw-r--r-- 1 root root 8123456 Aug 11 03:42 /boot/vmlinuz-6.1.0-25-amd64
-rw-r--r-- 1 root root 8098234 Jul 02 11:20 /boot/vmlinuz-6.1.0-23-amd64

The latent change

The trap in step 3 is assuming that a change takes effect when it is made. On Linux it frequently does not.

$ systemctl show myapp.service -p NeedDaemonReload; systemctl list-units --state=running --no-pager | wc -l; [ -f /var/run/reboot-required ] && cat /var/run/reboot-required

NeedDaemonReload=yes
64
*** System restart required ***

Validating the configuration that has not been applied yet is a read-only way to find out whether the next restart will succeed:

findmnt --verify --verbose
nginx -t
sshd -t
visudo -c
systemd-analyze verify /etc/systemd/system/myapp.service

Ruling change out

Sometimes nothing changed, and establishing that is a real finding - it eliminates a whole class of hypothesis. But it has to be established, not assumed, and it has to be stated precisely.

• Weak: “nothing changed.”
• Strong: “no package operations, no /etc modifications, no daemon-reload and no reboot on web02 between 06:00 and 09:15 UTC; the last Ansible run was 07:00 and converged with no changes; the deploy pipeline shows no release since Friday.”

The strong version tells the next person exactly which sources were checked and over what window, so they do not spend twenty minutes re-checking them. It also makes the remaining possibilities explicit: if nothing on the host changed, then either the load changed, something upstream changed, or something reached a threshold it had been approaching all along.

Knowledge check