Pre-checks and state capture - what to record before you touch anything

Half the incidents attributed to a change were not caused by it. The host had a failed unit before anyone logged in, or a filesystem at 98%, or a service that had been quietly restarting every ten minutes since Tuesday. The change arrived, the symptom became visible, and the change got the blame - and the rollback, which of course did not help.

Pre-checks exist to answer one question before you start: was this host healthy a minute ago? If you cannot answer it, you cannot interpret anything that happens next.

Two jobs, one pass

The pre-check pass does two different things and it is worth being clear which is which.

• Gating: proving the host is in a state where the change is safe to attempt. A gate can fail, and a failed gate stops the change.
• Capture: recording the before picture, so that afterwards you can compare rather than judge. Capture cannot fail; it just has to happen before you disturb anything.

The change management checklist in this course lists the gates as items to confirm. This lesson is about running them and keeping the output, because a checklist tick is an assertion while a command’s output is evidence with a timestamp.

The gates

$ systemctl is-system-running; systemctl list-units --state=failed --no-legend

degraded
● chrony.service loaded failed failed chrony, an NTP client/server

degraded is the most valuable single word in a pre-check. It means something was already broken, and it means that any symptom you see after the change has at least two candidate explanations.

$ df -h --output=pcent,ipcent,avail,target -x tmpfs -x devtmpfs -x squashfs

Use% IUse%  Avail Mounted on
47%    4%    62G /
91%   12%   3.2G /var

$ journalctl -p err -b --since '24 hours ago' --no-pager | wc -l; systemctl list-timers --all --no-pager | head -5

3
NEXT                        LEFT      LAST                        PASSED    UNIT
Tue 2026-08-11 22:00:00 UTC 6h left   Mon 2026-08-10 22:00:00 UTC 18h ago   backup.timer
Tue 2026-08-11 21:15:00 UTC 5h left   Tue 2026-08-11 09:15:00 UTC 6h ago    apt-daily.timer
Wed 2026-08-12 06:00:00 UTC 14h left  Tue 2026-08-11 06:00:00 UTC 9h ago    logrotate.timer

The remaining gates are less about the host and more about the recovery path.

• Out-of-band console answers, tested today - not assumed from the inventory
• The rollback artefact exists and is fetchable: the previous package, image, or snapshot
• Monitoring is up and not silenced for these hosts, so a regression is visible
• The configuration under change is in version control with no uncommitted local edits
• A second person knows the change is starting and when it is expected to end

Drift is a gate, not a detail

A change reviewed against the repository is a change reviewed against a host that may not exist. If somebody fixed something by hand at 03:00 last quarter and never committed it, your change is being applied to a configuration nobody has read, and your rollback will restore the committed state - silently discarding their fix.

$ sudo git -C /etc status --short; sudo git -C /etc log -1 --format='%h %cd %s'

 M nginx/conf.d/upstream.conf
M sysctl.d/99-tuning.conf
a41f0c2 Tue Jul 14 09:12:44 2026 +0000 base role apply

Where etckeeper is not installed, the package manager can still tell you which packaged files have been modified:

# Debian family - requires the debsums package
sudo debsums -ce

# RHEL family - built in
rpm -Va --nofiles --noscripts 2>/dev/null | head

Both list configuration files whose contents no longer match the package. That list is your drift inventory on a host with no configuration management history.

The capture

Capture is cheap, read-only, and the difference between “it looks fine” and “it matches”. Write it into one timestamped directory so the whole set can be handed to somebody else.

$ CHG=/var/tmp/chg-$(date -u +%Y%m%dT%H%M%SZ); mkdir -p "$CHG"; systemctl list-units --state=failed --no-legend > "$CHG/failed.txt"; systemctl list-unit-files --state=enabled --no-legend > "$CHG/enabled.txt"; dpkg -l > "$CHG/packages.txt" 2>/dev/null || rpm -qa | sort > "$CHG/packages.txt"; ss -tlnp > "$CHG/listeners.txt"; ip -br addr > "$CHG/addr.txt"; ip route > "$CHG/route.txt"; df -h > "$CHG/df.txt"; sysctl -a > "$CHG/sysctl.txt" 2>/dev/null; echo "$CHG"

/var/tmp/chg-20260811T154207Z

The single highest-value item in that set is the listener list. After the change, one diff answers the question that actually matters:

$ ss -tlnp > /var/tmp/after-listeners.txt; diff "$CHG/listeners.txt" /var/tmp/after-listeners.txt

5c5
< LISTEN 0  511  0.0.0.0:443  0.0.0.0:*  users:(("nginx",pid=1204,fd=7))
---
> LISTEN 0  511    127.0.0.1:443  0.0.0.0:*  users:(("nginx",pid=8891,fd=7))

Make the capture part of the procedure

A pre-check performed only when the operator remembers is a pre-check performed on the low-risk changes and skipped on the tense ones. Put it in the runbook as step zero, keep its output next to the change record, and gate the change on its exit status where you can:

set -e
systemctl is-system-running >/dev/null    # non-zero unless "running"
test "$(systemctl list-units --state=failed --no-legend | wc -l)" -eq 0
test "$(df --output=pcent /var | tail -1 | tr -dc '0-9')" -lt 85
test -z "$(git -C /etc status --porcelain)"
echo "pre-checks passed"

Anything that exits non-zero stops the change before it starts, which is the cheapest place for a change to stop.

Knowledge check