Lab: Canary-first patch run with an induced failure on host 3

Objective

By the end of this lab you will have run a real patch against five hosts with a one-host canary, watched it stop when the third host failed, and produced — from evidence rather than from the recap — a classification of all five hosts into patched, failed, and never attempted. You will then resume the rollout without re-patching anything that already succeeded.

Architecture

Five managed nodes patched in batches of 1, 2 and 2. One of them has a deliberately induced fault.

controller
    │  serial: [1, 2, 2]
    ├── batch 1: node1              canary
    ├── batch 2: node2, node3       node3 fails here
    └── batch 3: node4, node5       never attempted

Requirements

A controller with ansible-core 2.21.x.
Five managed nodes with a real package manager and systemd as PID 1. This lab installs and upgrades packages and restarts services. A container cannot do the service half honestly, and a container’s package database is not the thing you are trying to learn about. B-nested, real VMs, no simulation path.
SSH key access and become on each node.
Network access to a package repository, or a local mirror.
No out-of-band access requirement: this lab does not touch SSH, the firewall, networking or the kernel. It can leave a package upgraded and a service restarted, which Cleanup addresses.

Scenario

Your change record says: patch five application servers, one first, then the rest in pairs, and stop if anything goes wrong.

Something goes wrong on the third host. The change board’s question the next morning is not “why did it fail” — that is easy. It is “which of the five servers are now running the new package version, and which are not”, and the answer must be an evidence trail, not a recollection.

Tasks

Task 1: Capture the pre-patch state

WORKDIR="$HOME/ansible-canary-lab"
mkdir -p "$WORKDIR/reports"
cd "$WORKDIR"

inventory.yml:

appservers:
  hosts:
    node1: {ansible_host: 192.0.2.11}
    node2: {ansible_host: 192.0.2.12}
    node3: {ansible_host: 192.0.2.13}
    node4: {ansible_host: 192.0.2.14}
    node5: {ansible_host: 192.0.2.15}
  vars:
    ansible_user: operator
    patch_package: rsync

# capture.yml
- name: Record the installed version on every host
  hosts: appservers
  become: true
  gather_facts: true
  tasks:
    - name: Read the package database
      ansible.builtin.package_facts:

    - name: Record the version of the package under patch
      ansible.builtin.copy:
        content: |
          host: {{ inventory_hostname }}
          captured: {{ ansible_date_time.iso8601 }}
          package: {{ patch_package }}
          installed: {{ ansible_facts.packages[patch_package][0].version
                        | default('NOT INSTALLED') }}
          kernel: {{ ansible_facts.kernel }}
        dest: "{{ playbook_dir }}/reports/pre-patch-{{ inventory_hostname }}.yml"
        mode: '0644'
      delegate_to: localhost
      become: false

Read-only / Safecontroller— Reads the package database on five nodes. Changes nothing.

$ ansible-playbook -i inventory.yml capture.yml && grep -h installed reports/pre-patch-*.yml

Every host must have a recorded version before you continue. A host with NOT INSTALLED will behave differently in the patch play and is worth noticing now rather than mid-run.

Task 2: Write the patch play

# patch.yml
- name: Patch the application servers, canary first
  hosts: appservers
  become: true
  gather_facts: true
  serial: [1, 2, 2]
  max_fail_percentage: 0

  vars:
    patch_package: rsync

  tasks:
    - name: Announce the batch
      ansible.builtin.debug:
        msg: "batch: {{ ansible_play_batch | join(', ') }}"
      run_once: true

    - name: Record the version before this host is patched
      ansible.builtin.package_facts:

    - name: Save the pre-task version as a fact
      ansible.builtin.set_fact:
        version_before: >-
          {{ ansible_facts.packages[patch_package][0].version
             | default('absent') }}

    - name: Apply the update
      ansible.builtin.package:
        name: "{{ patch_package }}"
        state: latest
      register: patch_result

    - name: Re-read the package database
      ansible.builtin.package_facts:

    - name: Record what this host now runs
      ansible.builtin.copy:
        content: |
          host: {{ inventory_hostname }}
          patched_at: {{ ansible_date_time.iso8601 }}
          package: {{ patch_package }}
          version_before: {{ version_before }}
          version_after: {{ ansible_facts.packages[patch_package][0].version
                            | default('absent') }}
          changed: {{ patch_result.changed }}
        dest: "{{ playbook_dir }}/reports/patched-{{ inventory_hostname }}.yml"
        mode: '0644'
      delegate_to: localhost
      become: false

    - name: Verify the package is functional after the update
      ansible.builtin.command: "{{ patch_package }} --version"
      register: post_check
      changed_when: false

    - name: Assert the post-patch check succeeded
      ansible.builtin.assert:
        that: post_check.rc == 0
        fail_msg: "{{ patch_package }} is not runnable after the update"
        success_msg: "{{ inventory_hostname }} patched and verified"

max_fail_percentage: 0 with serial: [1, 2, 2]

max_fail_percentage: 0 means any failure in a batch aborts the play — because the comparison is strictly greater than, and any non-zero percentage exceeds zero.

Combined with serial: [1, 2, 2] that is a strict but coherent policy: the canary must pass before pair one, and pair one must pass before pair two. There is no tolerance anywhere, which for a patch run is usually correct — a package that fails to install on one host will very likely fail on the next.

The alternative, tolerating one failure per pair, is written max_fail_percentage: 50 and means “keep going even though half the batch broke”. Choose deliberately; do not inherit.

Also note the record-writing task: it writes to the controller (delegate_to: localhost) immediately after each host is patched, before the verification. That ordering matters. If the verification is what fails, you still have a record saying the host was patched — which is exactly the host state you need to know about.

Task 3: Induce the failure on node3

# induce.yml
- name: Hold the package manager lock on node3
  hosts: node3
  become: true
  gather_facts: false
  tasks:
    - name: Take the dpkg lock in the background
      ansible.builtin.shell: |
        set -euo pipefail
        nohup flock /var/lib/dpkg/lock-frontend sleep 1800 >/dev/null 2>&1 &
        echo $!
      args:
        executable: /bin/bash
      register: holder
      changed_when: true

    - name: Record the holder PID so cleanup can find it
      ansible.builtin.copy:
        content: "{{ holder.stdout | trim }}\n"
        dest: "{{ playbook_dir }}/reports/lock-holder-pid.txt"
        mode: '0644'
      delegate_to: localhost
      become: false

Service impact possiblecontroller— Blocks all package management on node3 for 30 minutes

$ ansible-playbook -i inventory.yml induce.yml && cat reports/lock-holder-pid.txt

Task 4: Run the patch and watch it stop

Service impact possiblecontroller— Upgrades a package on up to three of five hosts, then aborts

$ ansible-playbook -i inventory.yml patch.yml

TASK [Announce the batch] ******************************************************
ok: [node1] => {"msg": "batch: node1"}
...
TASK [Assert the post-patch check succeeded] ***********************************
ok: [node1] => {"msg": "node1 patched and verified"}

PLAY [Patch the application servers, canary first] *****************************
TASK [Announce the batch] ******************************************************
ok: [node2] => {"msg": "batch: node2, node3"}

TASK [Apply the update] ********************************************************
ok: [node2]
fatal: [node3]: FAILED! => {"changed": false, "msg": "Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 4471 (flock)"}

NO MORE HOSTS LEFT *************************************************************

PLAY RECAP *********************************************************************
node1                      : ok=8    changed=1    unreachable=0    failed=0
node2                      : ok=8    changed=1    unreachable=0    failed=0
node3                      : ok=3    changed=0    unreachable=0    failed=1

Illustrative output

Record the exit code — it is part of the change record:

ansible-playbook -i inventory.yml patch.yml > reports/patch-run.txt 2>&1
echo "exit=$?" | tee -a reports/patch-run.txt

A play aborted by max_fail_percentage exits 2, the same as any task failure. There is no distinct exit code for “stopped by policy”, which is worth knowing when a wrapper script is deciding what to do next.

Task 5: Classify all five hosts

The recap has three lines. The inventory has five hosts. Produce the classification the change board needs:

cd "$HOME/ansible-canary-lab"

# Hosts the run reached
sed -n '/PLAY RECAP/,$p' reports/patch-run.txt \
  | grep -oE '^node[0-9]+' | sort -u > reports/attempted.txt

# Hosts that failed, read from the recap lines only
sed -n '/PLAY RECAP/,$p' reports/patch-run.txt \
  | awk '/failed=[1-9]/ {print $1}' | sort -u > reports/failed.txt

# Hosts the inventory contains
ansible -i inventory.yml appservers --list-hosts \
  | tail -n +2 | tr -d ' ' | sort > reports/all.txt

echo "=== NEVER ATTEMPTED ==="
comm -13 reports/attempted.txt reports/all.txt

Read-only / Safecontroller— The hosts absent from the recap entirely

$ comm -13 reports/attempted.txt reports/all.txt

node4
node5

Illustrative output

Then confirm each classification against the hosts themselves, because the run output is a claim and the package database is the fact:

# classify.yml
- name: Establish the true post-abort state
  hosts: appservers
  become: true
  gather_facts: true
  tasks:
    - name: Read the package database
      ansible.builtin.package_facts:

    - name: Load the pre-patch record
      ansible.builtin.include_vars:
        file: "{{ playbook_dir }}/reports/pre-patch-{{ inventory_hostname }}.yml"
        name: pre
      delegate_to: localhost
      become: false

    - name: Report the classification
      ansible.builtin.debug:
        msg: >-
          {{ inventory_hostname }}:
          before={{ pre.installed }}
          now={{ ansible_facts.packages[patch_package][0].version
                 | default('absent') }}
          state={{ 'PATCHED' if ansible_facts.packages[patch_package][0].version
                                != pre.installed else 'UNPATCHED' }}

The recap and the fleet are different things

Three facts, and they do not line up the way people assume:

node1 and node2 are patched. The recap says so and the package database agrees.
node3 failed and is unpatched. The lock prevented the operation entirely, so it is in a clean prior state — which is the good kind of failure.
node4 and node5 were never attempted. They appear nowhere in the recap, they are unpatched, and nothing about their state is abnormal.

The dangerous case is a fourth category this lab does not produce: a host where the package operation partially succeeded. dpkg interrupted mid-transaction leaves a package in a half-configured state that package_facts reports as installed at the new version while the service is broken. The classification play must therefore always check functionality, not only version — which is why the patch play’s last task runs the binary.

Task 6: Write the change record entry

cd "$HOME/ansible-canary-lab"

cat > reports/change-record.md <<'MD'
# Patch run — CHG-XXXX

Policy: serial [1, 2, 2], max_fail_percentage 0
Package: rsync

| Host  | Before   | After    | State          | Evidence |
|-------|----------|----------|----------------|----------|
| node1 |          |          | PATCHED        | reports/patched-node1.yml |
| node2 |          |          | PATCHED        | reports/patched-node2.yml |
| node3 |          |          | FAILED         | reports/patch-run.txt |
| node4 |          |          | NOT ATTEMPTED  | absent from recap |
| node5 |          |          | NOT ATTEMPTED  | absent from recap |

Abort cause:
Remediation:
Resume plan:
MD

ls reports/

Fill it in from the artefacts. Every row must cite a file.

Task 7: Resume without re-patching

Fix the underlying cause first — release the lock — then resume against only the hosts that still need it:

cd "$HOME/ansible-canary-lab"

# Build the limit file from the classification, not from memory
{
  echo "# resume after CHG-XXXX abort, generated $(date -Is)"
  echo "# excludes hosts confirmed PATCHED"
  cat reports/failed.txt
  comm -13 reports/attempted.txt reports/all.txt
} > reports/resume.limit

cat reports/resume.limit

Resume is a new change, not a continuation

The temptation is to re-run patch.yml unchanged and let the idempotent package task skip the hosts already at the new version. It would work, and it is still the wrong instinct.

A resume run has a different blast radius from the original: three hosts, not five. It should have its own limit file, its own record, and its own canary — because the reason the first run stopped may or may not have been fixed, and finding out on one host is cheaper than on three.

ansible-playbook -i inventory.yml patch.yml \
  --limit @reports/resume.limit --list-hosts

Check that list before running it. Then run it.

Task 8: Release the induced fault

# release.yml
- name: Release the package manager lock on node3
  hosts: node3
  become: true
  gather_facts: false
  tasks:
    - name: Read the recorded holder PID
      ansible.builtin.slurp:
        src: "{{ playbook_dir }}/reports/lock-holder-pid.txt"
      register: pidfile
      delegate_to: localhost
      become: false

    - name: Kill the recorded holder
      ansible.builtin.command: "kill {{ (pidfile.content | b64decode) | trim }}"
      register: killed
      failed_when: false
      changed_when: killed.rc == 0

    - name: Find any remaining holder, in case the PID was stale
      ansible.builtin.shell: |
        fuser -k /var/lib/dpkg/lock-frontend 2>/dev/null || true
      args:
        executable: /bin/bash
      changed_when: false

    - name: Confirm the lock is free
      ansible.builtin.command: apt-get check
      register: aptcheck
      changed_when: false

Configuration changecontroller— Kills the lock holder on node3 and verifies apt works again

$ ansible-playbook -i inventory.yml release.yml

Then run the resume and confirm all five hosts reach the new version.

Validation

reports/pre-patch-node{1..5}.yml exist and record a version for each host, written before any patching.
The patch run’s recap contains exactly three host lines, and exits 2.
comm -13 reports/attempted.txt reports/all.txt names node4 and node5.
classify.yml reports PATCHED for node1 and node2 and UNPATCHED for node3, node4 and node5.
node3’s failure message names Could not get lock and a PID.
After release.yml, apt-get check on node3 exits 0.
The resume run with --limit @reports/resume.limit targets exactly three hosts, and after it every host reports the same version.
reports/change-record.md has an evidence file cited for every row.

Expected Outcome

ansible-canary-lab/
├── capture.yml, patch.yml, induce.yml, release.yml, classify.yml
├── inventory.yml
└── reports/
    ├── all.txt, attempted.txt, failed.txt
    ├── change-record.md
    ├── lock-holder-pid.txt
    ├── patch-run.txt
    ├── patched-node{1,2}.yml
    ├── pre-patch-node{1..5}.yml
    └── resume.limit

Five hosts at the same package version, an evidence trail showing the order in which they got there, and a change record that distinguishes failed from never-attempted without relying on anybody’s memory.

Troubleshooting

The canary batch is not honoured. serial: [1, 2, 2] must be a YAML list. Written as the string "1, 2, 2" it is not parsed as one and the whole play runs as a single batch — with no error. Confirm with the Announce the batch task’s output before trusting the policy.

max_fail_percentage: 0 did not abort. It has no effect without serial. Also confirm it is a number, not a quoted string.

package: state: latest reports changed on every host, every run. That is latest doing its job: it compares against the repository. For a patch play that is what you want; for a convergence play it is not, and the two should not be the same play.

The lock injection does not cause a failure. flock may not be present, or the path differs on a non-Debian host. On RHEL family the equivalent lock is /var/run/dnf.pid or a dnf transaction; use dnf -y install in a background loop instead, or induce the failure with a dpkg --set-selections hold on the package.

node4 and node5 appear in the recap as ok=0. Then the play did attempt them, which means the abort did not happen where you thought. Check that max_fail_percentage was actually applied — this is the failure mode the Announce the batch task exists to make visible.

A host reports the new version but the binary does not run. The half-configured dpkg case. dpkg --configure -a on that host, then re-run the verification task. This is the state that makes “check the version” an insufficient classification.

Cleanup

This lab upgraded a package on up to five hosts, held a package manager lock, and left report files. Restoring the package version is the part that may not be fully possible, and the cleanup says so rather than pretending.

Step 1. Release any remaining lock, unconditionally:

cd "$HOME/ansible-canary-lab"
ansible-playbook -i inventory.yml release.yml
ansible -i inventory.yml appservers -b -m command -a 'apt-get check'

Step 2. Decide what “restore” means for the package. Read the capture:

grep -h -A1 'package:' reports/pre-patch-*.yml

A package downgrade may be impossible, and forcing it is worse

If every host recorded the same version and you want it back, a downgrade is possible only while that version is still in a repository or in /var/cache/apt/archives. Check before attempting it:

# Substitute your own values before running:
PACKAGE=rsync
VERSION=3.2.7-1

ansible -i inventory.yml appservers -b -m shell \
  -a "apt-cache madison $PACKAGE | grep -c '$VERSION' || true"

If the version is not available, stop. Do not use --force-downgrades against a package pulled from a different source, and do not install a .deb from an unverified location to get an old version back. An upgraded package is a normal, supported state; a forcibly downgraded one with unsatisfied dependencies is not.

Record in your notes that the hosts remain at the upgraded version and why. That is an honest cleanup outcome for a patch lab, and it is the reason the Requirements ask for disposable VMs.

Where a snapshot exists, rolling the VM back is the correct restoration and is preferable to any package surgery.

Step 3. Confirm no lock holder survives on any host:

ansible -i inventory.yml appservers -b -m shell \
  -a 'fuser -v /var/lib/dpkg/lock-frontend 2>&1 || echo "lock free"'

ansible -i inventory.yml appservers -b -m shell \
  -a 'pgrep -a "flock" || echo "no flock processes"'

Step 4. Keep the change record and remove the working directory:

rm -rf deletes the whole lab directory, including anything you added

The next block removes ~/ansible-canary-lab and everything under it, recursively and without confirmation. If you saved notes, an inventory you want to keep, or a playbook you extended, copy it out first — the step above does that for the lab’s own deliverables and knows nothing about yours.

The path is written in full and anchored to $HOME deliberately. A relative rm -rf run from the wrong directory is how people delete work they meant to keep, and the shell gives no second chance.

Confirm what you are about to delete before you delete it:

ls -la "$HOME/ansible-canary-lab"

mkdir -p "$HOME/ansible-lab-deliverables/canary"
cp -a reports/change-record.md reports/resume.limit patch.yml \
      "$HOME/ansible-lab-deliverables/canary/"

rm -rf "$HOME/ansible-canary-lab"

Under the hood

The linear strategy — the default — runs one task across every host in the current batch before starting the next task. Under serial, the play is executed once per batch, which is why the PLAY [...] banner reappears between batches and why run_once fires once per batch rather than once per play.

At the end of each batch the strategy evaluates the failure policy. For max_fail_percentage, it compares failed hosts in that batch against the batch size; if the percentage is strictly greater than the threshold, it prints NO MORE HOSTS LEFT and stops scheduling further batches.

NO MORE HOSTS LEFT appearing twice, as in the output above, is normal and is worth recognising: it is emitted once per remaining play phase — the task loop and the handler flush — rather than once per abort.

The hosts in unscheduled batches are never instantiated as play hosts at all, which is why they contribute nothing to the recap. The recap is assembled from the stats callback, and the stats callback only ever sees a host that a task was dispatched to. There is no bookkeeping anywhere of “hosts this play was going to reach”, which is why the comm against --list-hosts is necessary rather than merely thorough.

What You Learned

A canary batch is serial: [1, ...] and nothing else is required — but it must be a YAML list, and a string form silently degrades to one batch.
max_fail_percentage: 0 means any failure aborts, because the comparison is strictly greater than zero.
The recap lists three hosts for a five-host inventory. Never attempted is not a state the recap can express, and comm against --list-hosts is how you recover it.
Write the per-host record before the verification task. If verification is what fails, you still need to know the host was patched.
A version match is not a health check. A half-configured package reports the new version and does not run, which is why the play executes the binary.
Resume is a new change with a new blast radius, its own limit file generated from the classification, and its own canary.
Some cleanups cannot fully restore. A package downgrade depends on the old version still existing; forcing one is worse than documenting that you did not.

Objectives

Prerequisites