Runbook: Patch a Linux fleet

Fleet patching is the routine change with the widest blast radius in most estates. It touches every host, it is scheduled rather than triggered, and the tooling makes it a one-line command. That combination is why it is also the change most likely to take out a whole tier simultaneously.

Two decisions do most of the work here. Pin the repository so every wave installs the same versions, and wave the rollout so a bad package meets five hosts rather than five hundred.

When to use this runbook

• Scheduled monthly or quarterly patching.
• An out-of-cycle security update.
• Bringing a drifted fleet back to a common package baseline.

Not for kernel updates that require a reboot as the point of the exercise - that is the rolling kernel update runbook, which this one hands off to.

Blast radius

Every host in scope. State it as a table before you start:

wave 0   staging          6 hosts    no user impact
wave 1   canary           3 hosts    1 per production tier
wave 2   web              40 hosts   batched 25%
wave 3   app              60 hosts   batched 25%
wave 4   db replicas      8 hosts    batched 1
wave 5   db primaries     4 hosts    one at a time, with failover

Every one of those numbers comes from --list-hosts against the exact command that will run. An estimate from reading the inventory file is where the “we thought that group had twelve hosts” incidents start.

Step 1: Pin the repository snapshot

This is the step that makes the rest reproducible.

- name: Point hosts at the pinned snapshot for this patch cycle
hosts: patch_scope
become: true
vars:
  patch_snapshot: '2026-08-01'
tasks:
  - name: Repository points at the frozen snapshot
    ansible.builtin.template:
      src: internal-mirror.repo.j2
      dest: /etc/yum.repos.d/internal-mirror.repo
      owner: root
      group: root
      mode: '0644'
    when: ansible_facts['os_family'] == 'RedHat'

  - name: Metadata cache reflects the pinned snapshot
    ansible.builtin.dnf:
      update_cache: true
    when: ansible_facts['os_family'] == 'RedHat'

ansible patch_scope -b -m command \
-a 'dnf repoinfo internal-mirror' -o | grep -iE 'Repo-baseurl|Repo-updated'

Step 2: Inventory what is installed

This is the rollback reference. Capture it before anything changes.

- name: Record the pre-patch package state
hosts: patch_scope
gather_facts: true
tasks:
  - name: Gather installed packages
    ansible.builtin.package_facts:
      manager: auto

  - name: Write a per-host manifest to the controller
    ansible.builtin.copy:
      content: |
        {% for name, versions in ansible_facts.packages.items() | sort %}
        {{ name }} {{ versions | map(attribute='version') | join(',') }}
        {% endfor %}
      dest: "./patch-baseline/{{ inventory_hostname }}.txt"
      mode: '0644'
    delegate_to: localhost
    become: false

Also record the free-space precondition, because a patch run that fills /boot fails halfway through a transaction:

ansible patch_scope -m command -a 'df -h /var /boot' -o
ansible patch_scope -b -m command -a 'systemctl is-system-running' -o

A host already in degraded state before patching should not be patched. Whatever is wrong there will be blamed on the patch.

Step 3: Enumerate what will change

# RHEL family - list, do not install
ansible patch_scope -b -m dnf -a 'list=updates' -o \
| tee updates-available.txt

# Debian family - simulate
ansible patch_scope -b -m command \
-a 'apt-get -s upgrade' -o | tee updates-available-deb.txt

Read that list for two things: packages whose update implies a service restart, and packages whose update implies a reboot. Kernel, glibc, systemd and openssl are the usual suspects, and they change the shape of the change.

Step 4: Wave 0 - non-production

ansible-playbook -i inventories/staging patch.yml --list-hosts
ansible-playbook -i inventories/staging patch.yml \
--diff | tee "patch-wave0-$(date -u +%Y%m%dT%H%M%SZ).log"
echo "exit=$?"

The patch play itself:

- name: Patch a wave
hosts: "{{ patch_wave }}"
become: true
serial: "{{ patch_batch | default('25%') }}"
max_fail_percentage: 0
tasks:
  - name: Enough free space to proceed
    ansible.builtin.assert:
      that:
        - ansible_facts['mounts'] | selectattr('mount', 'equalto', '/var')
          | map(attribute='size_available') | first > 2147483648
      fail_msg: "Less than 2 GiB free on /var - refusing to patch"

  - name: Apply all available updates from the pinned snapshot
    ansible.builtin.dnf:
      name: '*'
      state: latest
      update_cache: false
    when: ansible_facts['os_family'] == 'RedHat'
    register: patch_result

  - name: Apply all available updates from the pinned snapshot
    ansible.builtin.apt:
      upgrade: safe
      update_cache: false
    when: ansible_facts['os_family'] == 'Debian'
    register: patch_result_deb

  - name: Service answers a real request after patching
    ansible.builtin.uri:
      url: "http://{{ ansible_host }}:8080/healthz"
      status_code: 200
      return_content: true
    register: health
    retries: 6
    delay: 10
    until: health.status == 200 and 'ok' in health.content
    when: "'web' in group_names"

The assert at the top is a refusal, not a warning. A host without space fails at task zero, before the transaction starts, which leaves it exactly as it was. That is a clean failure you can act on; a part-completed dnf transaction on a full disk is not.

The health check at the bottom is what makes max_fail_percentage: 0 meaningful. Without it, the play succeeds as long as the packages installed, whether or not the service came back.

Hold wave 0 overnight if the cycle allows. Most package regressions that matter show up under a full day of use, not in the ten minutes after the transaction.

Step 5: Wave 1 - one production host per tier

ansible-playbook -i inventories/production patch.yml \
-e patch_wave=canary --list-hosts
ansible-playbook -i inventories/production patch.yml \
-e patch_wave=canary --diff | tee "patch-wave1.log"

One host per tier, because a package that breaks the web tier and a package that breaks the database tier are different packages, and a canary drawn only from web tells you nothing about db.

Hold, and watch the same things a canary deployment watches: error rate on those specific hosts, latency, log volume, memory.

Step 6: The gate

Before widening, answer three questions in writing:

1. Did every wave 1 host complete the transaction cleanly?
2. Is every service on those hosts healthy against a real request?
3. Does any host now require a reboot - and is that in scope?

# RHEL family
ansible patch_scope -b -m command -a 'needs-restarting -r' -o

# Debian family
ansible patch_scope -b -m stat -a 'path=/var/run/reboot-required' -o \
| grep -c '"exists": true'

Step 7: Remaining waves

for wave in web app db_replica; do
ansible-playbook -i inventories/production patch.yml \
  -e "patch_wave=$wave" --list-hosts | tee "listhosts-$wave.txt"
read -r -p "Proceed with $wave? [y/N] " ok
[ "$ok" = "y" ] || break
ansible-playbook -i inventories/production patch.yml \
  -e "patch_wave=$wave" --diff | tee "patch-$wave.log"
done

The read is deliberate. A loop that runs every wave unattended is a loop that patches the database tier at 3am after the web tier failed at 2:55am, and the batch gate inside the play does not stop the next invocation.

Database primaries get serial: 1 and their own procedure. Patching two primaries at once is how a patch window becomes a data incident.

Step 8: Verify per wave

Do not verify from the run output. The run output tells you what Ansible believed; package_facts tells you what is installed.

- name: Confirm the patch landed
hosts: "{{ patch_wave }}"
gather_facts: false
tasks:
  - name: Gather installed packages
    ansible.builtin.package_facts:
      manager: auto

  - name: Report the version of a package that was expected to change
    ansible.builtin.debug:
      msg: >-
        {{ inventory_hostname }} openssl
        {{ ansible_facts.packages['openssl'] | map(attribute='version') | join(',')
           if 'openssl' in ansible_facts.packages else 'ABSENT' }}

# Nothing left in a failed state
ansible patch_scope -b -m command -a 'systemctl --failed --no-pager --plain' -o

# The recap host list must match what --list-hosts said before the run
grep -oE '^[a-z0-9.-]+ +:' "patch-web.log" | tr -d ' :' | sort > recap-web.txt
grep -oE '[a-z0-9.-]+\.example\.com' listhosts-web.txt | sort > intended-web.txt
diff intended-web.txt recap-web.txt

That diff is the check that catches the silent case. If a batch failed and the play stopped, the untouched hosts are absent from the recap entirely - not marked skipped, not marked failed, simply not there. Verified on 2.21.3. An empty diff means every intended host was attempted; a non-empty diff is your list of hosts still unpatched.

Rollback

Establish per package, before wave 0, whether rollback is possible.

# What transaction did we just run?
ansible patch_scope -b -m command -a 'dnf history list --reverse' -o | tail -5

# What did it contain?
ansible web01.example.com -b -m command -a 'dnf history info last' -o

ansible-playbook -i inventories/production rollback-patch.yml \
--limit web01.example.com -e transaction_id=42 --list-hosts

ansible web01.example.com -b -m command -a 'dnf history undo 42 -y'
ansible web01.example.com -b -m command -a 'dnf history info last'

dnf history undo only works while the previous package versions are still reachable - in the repository, the local cache, or a snapshot mirror. Pinning to a frozen snapshot helps here too: the old versions are still there.

On Debian family there is no transaction undo. Rollback means downgrading to a specific version, and that version must still be in the archive:

ansible web01.example.com -b -m apt \
-a 'name=nginx=1.24.0-2~deb12u1 state=present allow_downgrade=true force_apt_get=true' -o

Roll back one wave at a time, verifying between. And stop the rollout first: a rollout and a rollback running against overlapping host sets produces a fleet in three states, which is harder to reason about than either failure.

Common patterns

Escalation

Escalate when:

• Wave 1 shows any service degradation. Continuing is the service owner’s decision.
• A package fails the same way on multiple hosts. That is a repository or dependency problem and widening spreads it.
• Rollback is not possible for a package that has caused a problem.
• The snapshot pin cannot be honoured everywhere.
• Hosts fail the free-space precondition. Skipping the check to make the window is how a transaction dies halfway.