Automated patching in a fleet - what to automate and what never to

The first four lessons in this part built a staged rollout: canary, waves, health validation between them, rollback if a wave regresses. Then somebody enables unattended upgrades across the fleet, and every host that is powered on at 06:00 applies the same change inside the same hour.

Both are defensible. They are not compatible by default, and the reconciliation is the subject of this lesson. The mechanics of configuring unattended-upgrades and dnf-automatic are covered in the packages part; what follows is the decision about where to point them and what to fence off.

What automation buys and what it costs

The first column is not marketing. Most real compromises exploit vulnerabilities that had a patch available for weeks, and the single largest cause of that is a manual process nobody had time for. Automation is usually the right default.

The second column is why it is a default and not a policy.

The decision, per tier

Automation is appropriate when three things hold: the change is small, the reversal is cheap, and the failure is detectable without a human. Score each tier of the fleet against those.

The line that matters is the last one. A database primary, a Corosync node, a Pacemaker-managed resource or an etcd member must not have a service restarted or a host rebooted by a timer that has no idea what quorum is. Two nodes of a three-node cluster rebooting inside the same hour is a quorum loss, and the timer will happily do it because both hosts are running the same configuration.

Making automation wave-aware

The way to have both is to stop treating the schedule as the thing that stages the rollout, and let the repository stage it instead.

Each tier points at a different snapshot of the same repository. Automation runs everywhere on the same schedule, but dev is consuming this week’s packages and production is consuming the set that dev validated a week ago. Promotion is a single change to which snapshot a tier resolves - not a change to any host.

upstream security repo
        |
        v
   snapshot 2026-08-11  <-- dev, CI      (automation on)
   snapshot 2026-08-04  <-- staging      (automation on)
   snapshot 2026-07-28  <-- production   (automation on)

This inverts the usual arrangement in a useful way. The waves are still waves, the canary is still a canary, and no host has a bespoke schedule. The next lesson covers how to build and promote those snapshots.

Until you have that, the poorer substitute is different schedules per tier, which works but couples the rollout to wall clock time and gives you no way to say “production installs exactly what staging installed”.

Guardrails

Splay

Debian’s timers already randomise, and the values are worth knowing because they are what stops a fleet stampede on your mirror:

$ systemctl cat apt-daily-upgrade.timer

# /usr/lib/systemd/system/apt-daily-upgrade.timer
[Unit]
Description=Daily apt upgrade and clean activities
After=apt-daily.timer

[Timer]
OnCalendar=*-*-* 6:00
RandomizedDelaySec=60m
Persistent=true

[Install]
WantedBy=timers.target

An hour of jitter spreads a few hundred hosts adequately. For thousands, widen it with a drop-in rather than editing the shipped unit:

sudo systemctl edit apt-daily-upgrade.timer
# [Timer]
# RandomizedDelaySec=6h

systemctl show apt-daily-upgrade.timer -p RandomizedDelaySec
systemctl list-timers apt-daily-upgrade.timer

Persistent=true deserves attention too: a host that was off at 06:00 runs the job as soon as it boots, jitter and all. That is correct behaviour for patching and surprising behaviour if you were relying on the window.

The RHEL-family equivalent is random_sleep in /etc/dnf/automatic.conf, plus the same RandomizedDelaySec on dnf-automatic.timer.

Scope

Restrict what automation is allowed to install. On Debian this is the origins pattern - security only, for anything above staging - plus an explicit blocklist for packages whose upgrade is a change you want to make deliberately:

grep -n -A8 'Unattended-Upgrade::Package-Blacklist' \
    /etc/apt/apt.conf.d/50unattended-upgrades

Kernels, the database server, the container runtime and anything with a config-file migration belong on that list. So does anything you have pinned, because a pin and a blocklist answer different questions and you want both.

Version holds are the coarser hammer and survive a configuration mistake:

sudo apt-mark hold postgresql-15
apt-mark showhold

# RHEL family
sudo dnf versionlock add postgresql-server
dnf versionlock list

Reboot policy

The dangerous default is not “reboot”, it is “reboot without knowing whether this host may reboot right now”.

// /etc/apt/apt.conf.d/50unattended-upgrades
Unattended-Upgrade::Automatic-Reboot "false";
Unattended-Upgrade::Automatic-Reboot-WithUsers "false";
Unattended-Upgrade::Automatic-Reboot-Time "02:00";

Leave Automatic-Reboot false above the dev tier and let the rolling-maintenance process own reboots, because that process is the only thing that knows about quorum, drain order and active sessions. The reboot-requirement lesson in the vulnerability part covers detecting that a reboot is pending; the point here is who decides to act on it.

Observability

Automation with no reporting is a process nobody is running. Three things to collect from every host:

# What was actually installed, and whether it errored
sudo tail -n 40 /var/log/unattended-upgrades/unattended-upgrades.log
sudo grep -c ERROR /var/log/unattended-upgrades/unattended-upgrades.log

# The transaction record, from the package manager itself
grep ' upgrade ' /var/log/dpkg.log | tail -20
sudo dnf history list | head

# Did the last run succeed, and when was it
systemctl status apt-daily-upgrade.service --no-pager
systemctl list-timers --all apt-daily-upgrade.timer

Turn the last one into a metric. “Time since last successful automated patch run” per host is the single number that catches the failure mode automation is worst at: a host where the timer has been failing silently for four months and everyone assumed it was covered.

Verify without changing anything before you trust a new configuration:

sudo unattended-upgrade --dry-run --debug

The kill switch

Before enabling automation on production, answer this: a bad package is in the repository right now, and hosts are picking it up. How do you stop the fleet in the next five minutes?

Have an answer that does not require touching hosts one at a time:

# Per host - fine for a handful, useless for a fleet
sudo systemctl mask --now apt-daily-upgrade.timer

# Fleet-wide, through configuration management, is the real answer
# ansible -i inventory all -m systemd \
#     -a "name=apt-daily-upgrade.timer enabled=no state=stopped masked=yes" -b

If your repository is snapshotted, the faster switch is to repoint the tier at the previous snapshot, which also fixes the hosts that already have the mirror index cached.

Test the kill switch on a schedule. An untested kill switch is a plan, and plans fail on the day the SSH bastion is one of the hosts that took the bad update.

Knowledge check