Lab: Partial fleet failure — diagnose, classify, decide · Lab · Ansible

Objective

By the end of this lab you will have run a change against ten hosts where two fail a task and three lose network connectivity mid-run, and you will have produced four classified lists — complete, failed, unreachable, never attempted — each backed by evidence gathered from the hosts rather than from the run output. You will then make and justify a different decision for each class.

Architecture

Ten managed nodes. Two carry a fault that makes a task fail; three have their inbound SSH blocked part-way through the run.

controller ──ssh──▶ node01 .. node10

  node03, node07   task fails      (a missing dependency)
  node05,06,08     SSH blocked mid-run  (nftables rule added)
  the rest         complete normally

Requirements

A controller with ansible-core 2.21.x.
Ten managed VMs with systemd as PID 1 and a working firewall. The connection-loss half of this lab needs genuine network interruption and real service state; neither is honest in a container. B-nested only.
SSH key access and become on all ten.
Out-of-band access to node05, node06 and node08. Task 4 blocks inbound SSH on those three. The rule is added by a self-cancelling mechanism with a timeout, and Cleanup removes it, but a mistake here locks you out over the network and the recovery is a console.
nftables on the managed nodes, or the iptables equivalent adapted.

Scenario

A configuration change ran against ten hosts overnight. The recap in the job log shows a mixture of ok, failed and unreachable, and the engineer who scheduled it is on leave.

You have to answer three questions before anything else can happen: which hosts got the change, which hosts are in an unknown state, and what to do about each. The recap is where you start and it is not where you finish.

Tasks

Task 1: Set up and capture

WORKDIR="$HOME/ansible-triage-lab"
mkdir -p "$WORKDIR/reports"
cd "$WORKDIR"

inventory.yml:

fleet:
  hosts:
    node[01:10]:
  vars:
    ansible_user: operator

Give each host an ansible_host in host_vars/, or use DNS. Then:

# setup.yml
- name: Install the helper the change depends on
  hosts: fleet
  become: true
  gather_facts: false
  tasks:
    - name: Install the config validator
      ansible.builtin.copy:
        content: |
          #!/bin/bash
          # Validates the rendered config. Exits non-zero if it is wrong.
          grep -q '^setting=' "$1" || exit 1
          exit 0
        dest: /usr/local/bin/validate-appconfig
        mode: '0755'

    - name: Record the pre-change state
      ansible.builtin.stat:
        path: /etc/appconfig
      register: pre

    - name: Save the capture
      ansible.builtin.copy:
        content: |
          host: {{ inventory_hostname }}
          appconfig_existed: {{ pre.stat.exists }}
        dest: "{{ playbook_dir }}/reports/pre-{{ inventory_hostname }}.yml"
        mode: '0644'
      delegate_to: localhost
      become: false

Task 2: The change play

# change.yml
- name: Apply the configuration change
  hosts: fleet
  become: true
  gather_facts: false

  tasks:
    - name: Render the configuration
      ansible.builtin.copy:
        content: |
          setting=new-value
          applied_by=ansible
        dest: /etc/appconfig
        mode: '0644'
        validate: '/usr/local/bin/validate-appconfig %s'

    - name: Wait, so there is time to interrupt the run
      ansible.builtin.wait_for:
        timeout: 45
      delegate_to: localhost
      become: false
      run_once: true

    - name: Register the change with the local audit log
      ansible.builtin.lineinfile:
        path: /var/log/appconfig-changes.log
        line: "{{ ansible_date_time.iso8601 | default('unknown') }} applied"
        create: true
        mode: '0644'

    - name: Verify the configuration is readable and correct
      ansible.builtin.command: /usr/local/bin/validate-appconfig /etc/appconfig
      changed_when: false

Task 3: Induce the task failure on node03 and node07

cd "$HOME/ansible-triage-lab"

ansible -i inventory.yml node03,node07 -b -m file \
  -a 'path=/usr/local/bin/validate-appconfig state=absent'

Now validate: on the copy task has no validator, and the task fails before writing anything. That is a clean failure — the host is unchanged — and recognising that class is half the triage.

Task 4: Block SSH on three hosts mid-run

# block-ssh.yml
- name: Block inbound SSH, with an automatic release
  hosts: node05,node06,node08
  become: true
  gather_facts: false

  tasks:
    - name: Confirm the at daemon is running before we depend on it
      ansible.builtin.command: systemctl is-active atd
      register: atd
      changed_when: false
      failed_when: atd.stdout != 'active'

    - name: Schedule the release FIRST, before installing the block
      ansible.builtin.shell: |
        set -euo pipefail
        echo 'nft delete table inet triagelab' | at now + 15 minutes
      args:
        executable: /bin/bash
      changed_when: true

    - name: Install the block
      ansible.builtin.shell: |
        set -euo pipefail
        nft add table inet triagelab
        nft add chain inet triagelab input '{ type filter hook input priority -10 ; }'
        nft add rule inet triagelab input tcp dport 22 ct state new drop
      args:
        executable: /bin/bash
      changed_when: true

Run the change and the block together, so the block lands while the change is in its 45-second wait:

cd "$HOME/ansible-triage-lab"

ansible-playbook -i inventory.yml change.yml > reports/change-run.txt 2>&1 &
CHANGE_PID=$!

sleep 10
ansible-playbook -i inventory.yml block-ssh.yml

wait "$CHANGE_PID"
echo "change run exit: $?"
tail -20 reports/change-run.txt

Task 5: Read the recap, and know what it does not say

Read-only / Safecontroller— The recap after a partial failure

$ sed -n '/PLAY RECAP/,$p' reports/change-run.txt

PLAY RECAP *********************************************************************
node01                     : ok=4    changed=2    unreachable=0    failed=0
node02                     : ok=4    changed=2    unreachable=0    failed=0
node03                     : ok=0    changed=0    unreachable=0    failed=1
node04                     : ok=4    changed=2    unreachable=0    failed=0
node05                     : ok=1    changed=1    unreachable=1    failed=0
node06                     : ok=1    changed=1    unreachable=1    failed=0
node07                     : ok=0    changed=0    unreachable=0    failed=1
node08                     : ok=1    changed=1    unreachable=1    failed=0
node09                     : ok=4    changed=2    unreachable=0    failed=0
node10                     : ok=4    changed=2    unreachable=0    failed=0

Illustrative output

Read the three classes carefully, because they mean different things.

unreachable=1 with changed=1 is the dangerous line

node05 shows ok=1 changed=1 unreachable=1.

That is a host which completed one task successfully, changed something, and then became unreachable. The configuration file was written. The audit log line was not. The verification never ran.

Contrast with node03: ok=0 changed=0 failed=1. That host failed its first task and changed nothing. It is in a clean, known state.

The two lines look similarly bad and are operationally opposite. A failed host is usually fine; an unreachable host that had already changed something is the one you cannot reason about.

unreachable also means Ansible could not establish or maintain a connection — it says nothing about whether the machine is running. A host that is powered off, one that is up but firewalled, and one whose sshd crashed all report identically.

Task 6: Build the four classified lists

The recap gives you three classes. Extract them mechanically:

cd "$HOME/ansible-triage-lab"
R=reports/change-run.txt

sed -n '/PLAY RECAP/,$p' "$R" | awk '/^node/ && /failed=0/ && /unreachable=0/ {print $1}' \
  | sort > reports/class-complete.txt

sed -n '/PLAY RECAP/,$p' "$R" | awk '/^node/ && /failed=[1-9]/ {print $1}' \
  | sort > reports/class-failed.txt

sed -n '/PLAY RECAP/,$p' "$R" | awk '/^node/ && /unreachable=[1-9]/ {print $1}' \
  | sort > reports/class-unreachable.txt

ansible -i inventory.yml fleet --list-hosts | tail -n +2 | tr -d ' ' | sort \
  > reports/class-all.txt

sed -n '/PLAY RECAP/,$p' "$R" | grep -oE '^node[0-9]+' | sort -u \
  > reports/class-attempted.txt

comm -13 reports/class-attempted.txt reports/class-all.txt \
  > reports/class-never-attempted.txt

wc -l reports/class-*.txt

Task 7: Establish the truth from the hosts

Verify every host independently, including the ones the recap says are fine. This is the step that separates triage from guesswork.

# establish-truth.yml
- name: Establish the actual state of every host
  hosts: fleet
  become: true
  gather_facts: false
  ignore_unreachable: true

  tasks:
    - name: Read the configuration file
      ansible.builtin.slurp:
        src: /etc/appconfig
      register: conf
      failed_when: false

    - name: Read the audit log
      ansible.builtin.slurp:
        src: /var/log/appconfig-changes.log
      register: auditlog
      failed_when: false

    - name: Report the true state
      ansible.builtin.debug:
        msg: >-
          {{ inventory_hostname }}:
          config={{ 'PRESENT' if conf.content is defined else 'ABSENT' }}
          audit={{ 'LOGGED' if auditlog.content is defined else 'MISSING' }}
      when: not (ansible_host_unreachable | default(false))

    - name: Record it
      ansible.builtin.copy:
        content: |
          host: {{ inventory_hostname }}
          reachable: {{ not (ansible_host_unreachable | default(false)) }}
          config_present: {{ conf.content is defined }}
          audit_logged: {{ auditlog.content is defined }}
        dest: "{{ playbook_dir }}/reports/truth-{{ inventory_hostname }}.yml"
        mode: '0644'
      delegate_to: localhost
      become: false

Run it while the three hosts are still blocked, and then again after the at job releases them, and compare:

ansible-playbook -i inventory.yml establish-truth.yml
cat reports/truth-node05.yml

Read-only / Safecontroller— node05 after the block expires: config written, audit line missing

$ cat reports/truth-node05.yml

host: node05
reachable: true
config_present: true
audit_logged: false

Illustrative output

That is the partial-application state, established as a fact. node05 has the new configuration and no record that it was applied — which is exactly the combination that makes a subsequent audit report it as an unmanaged change.

Task 8: Decide, per class

Write reports/triage.md. Four classes, four decisions, each with a justification.

Class	Hosts	State	Decision
Complete	node01,02,04,09,10	config + audit line	verify and close
Failed clean	node03, node07	unchanged	retry after fixing the cause
Unreachable, partially applied	node05, 06, 08	config, no audit line	investigate then re-run
Never attempted	—	untouched	retry in the resume run

The four responses, and when each is right

Retry is right when the failure is understood, host-specific and fixed. node03 and node07 lost a helper; restore it and re-run. The hosts changed nothing, so a retry has the same blast radius as the original run.

Rollback is right when the change is partially applied across enough of the fleet that a mixed state is worse than no change. Nothing here needs it, because the config change is compatible with the unchanged state — but that is a judgement about this change, and it must be made explicitly.

Quarantine is right when a host cannot be brought to a known state now and must be prevented from serving traffic or receiving further automation. A host that has been unreachable for a day, with an unknown partial change, belongs out of the load balancer and out of the inventory group until somebody looks at it.

Investigate is right when you do not yet know which of the other three applies. node05, node06 and node08 are here initially: they went unreachable simultaneously, which is a pattern, and a pattern means a common cause you have not identified. Three hosts failing independently is bad luck; three failing together is a change somebody made.

The mistake to avoid is jumping to retry for everything because retry is the easy one. A retry against a host in an unknown partial state can be a second partial change on top of the first.

Task 9: Resume without re-running what completed

cd "$HOME/ansible-triage-lab"

# Fix the cause on the failed hosts
ansible-playbook -i inventory.yml setup.yml --limit node03,node07

# Build the resume limit from the classification
{
  echo "# resume after partial failure, $(date -Is)"
  echo "# excludes hosts verified complete"
  cat reports/class-failed.txt
  cat reports/class-unreachable.txt
  cat reports/class-never-attempted.txt
} | sort -u > reports/resume.limit

ansible-playbook -i inventory.yml change.yml \
  --limit @reports/resume.limit --list-hosts

Check the list, then run it, then re-run establish-truth.yml and confirm all ten hosts report config_present: true and audit_logged: true.

Validation

The recap shows failed=1 for node03 and node07 with changed=0, and unreachable=1 with changed=1 for node05, node06 and node08.
reports/class-*.txt partition the ten hosts with no host in two lists and none missing.
establish-truth.yml completes on all ten hosts even while three are unreachable, because of ignore_unreachable: true.
reports/truth-node05.yml reports config_present: true and audit_logged: false once the host is reachable again.
reports/truth-node03.yml reports config_present: false — the clean failure.
reports/triage.md assigns a decision and a justification to each of the four classes.
After the resume, all ten truth-*.yml report both true.
nft list ruleset on node05, node06 and node08 shows no triagelab table.

Expected Outcome

ansible-triage-lab/
├── block-ssh.yml, change.yml, establish-truth.yml, setup.yml
├── inventory.yml
└── reports/
    ├── change-run.txt
    ├── class-{all,attempted,complete,failed,never-attempted,unreachable}.txt
    ├── pre-node{01..10}.yml
    ├── resume.limit
    ├── triage.md
    └── truth-node{01..10}.yml

Ten hosts in a verified, identical state, and a written triage that distinguishes four classes and justifies a decision for each.

Troubleshooting

Every host reports unreachable. The block was applied to the wrong group, or your controller’s own connection to the fleet went through one of the blocked hosts as a bastion. Check ansible_ssh_common_args in the inventory.

The block does not expire. atd was not running when the at job was submitted, so the job was queued and never scheduled, or it was never accepted at all. The task asserting systemctl is-active atd exists to prevent this — if you skipped it, go to the console.

ignore_unreachable does not seem to work. It must be on the play or the task. It changes what happens after a host becomes unreachable; it does not make the connection succeed, so tasks against that host still produce no data. That is expected — the value is that the play completes.

ansible_host_unreachable is undefined. It is set only on hosts that actually went unreachable. | default(false) is required, as used above.

slurp fails on a file that exists. slurp base64-encodes the whole file into a fact, so it fails on large files and on files the become-user cannot read. For a large log, use command: tail -5 with changed_when: false instead.

A host reports config_present: true but the content is wrong. The truth play checks presence, not content. Extend it — conf.content | b64decode and a regex_search for the expected value — because “the file exists” is a weaker claim than most triage needs.

The recap has fewer than ten lines. Some hosts were never attempted, which on a play with no serial means the play aborted early. Check for any_errors_fatal or a max_fail_percentage you did not intend.

Cleanup

This lab blocked SSH on three hosts, removed a helper from two, and wrote a configuration file on up to ten. All of it must be reversed, and the firewall rule is the one that must be reversed first.

Step 1. Remove the firewall block, and verify. Do this before anything else, and do not assume the at job ran:

# unblock-ssh.yml
- name: Remove the triage lab firewall block
  hosts: node05,node06,node08
  become: true
  gather_facts: false

  tasks:
    - name: Remove the table if it exists
      ansible.builtin.shell: |
        set -uo pipefail
        nft list table inet triagelab >/dev/null 2>&1 && nft delete table inet triagelab
        exit 0
      args:
        executable: /bin/bash
      changed_when: true

    - name: Confirm the table is gone
      ansible.builtin.shell: |
        set -uo pipefail
        nft list table inet triagelab >/dev/null 2>&1 && echo PRESENT || echo ABSENT
      args:
        executable: /bin/bash
      register: tbl
      changed_when: false

    - name: Assert the block is removed
      ansible.builtin.assert:
        that: tbl.stdout | trim == 'ABSENT'
        fail_msg: >-
          The triagelab table is still present on {{ inventory_hostname }}.
          Remove it from the console before this session ends.

    - name: Remove any queued at jobs the lab created
      ansible.builtin.shell: |
        set -uo pipefail
        for j in $(atq | awk '{print $1}'); do
          at -c "$j" 2>/dev/null | grep -q 'triagelab' && atrm "$j"
        done
        exit 0
      args:
        executable: /bin/bash
      changed_when: true

Verify SSH from a NEW connection before you finish

Removing the rule from within an existing session proves nothing about whether new connections work — the whole point of the rule was that it dropped ct state new and left established sessions alone.

Open a fresh connection and confirm:

# Substitute your own values before running:
HOST=node05

ssh -o ConnectTimeout=10 -o BatchMode=yes "operator@$HOST" 'echo reachable'

Do not close your working session until that succeeds on all three hosts.

Step 2. Restore the helper on the two failed hosts and remove the lab’s files everywhere:

# cleanup.yml
- name: Remove everything the lab created
  hosts: fleet
  become: true
  gather_facts: false

  tasks:
    - name: Load the pre-lab capture
      ansible.builtin.include_vars:
        file: "{{ playbook_dir }}/reports/pre-{{ inventory_hostname }}.yml"
        name: pre
      delegate_to: localhost
      become: false

    - name: Refuse to continue without a capture
      ansible.builtin.assert:
        that: pre.host is defined
        fail_msg: "No pre-lab capture for {{ inventory_hostname }}."

    - name: Remove the config file the lab created
      ansible.builtin.file:
        path: /etc/appconfig
        state: absent
      when: not (pre.appconfig_existed | bool)

    - name: Remove the validator and the audit log
      ansible.builtin.file:
        path: "{{ item }}"
        state: absent
      loop:
        - /usr/local/bin/validate-appconfig
        - /var/log/appconfig-changes.log

Step 3. Verify the fleet is clean and reachable:

cd "$HOME/ansible-triage-lab"

ansible -i inventory.yml fleet -m ping
ansible -i inventory.yml node05,node06,node08 -b -m shell \
  -a 'nft list ruleset | grep -c triagelab || echo 0'
ansible -i inventory.yml fleet -b -m stat -a 'path=/etc/appconfig' | grep -c '"exists": true' || true

All ten must respond to ping, no host may report a triagelab rule, and atq must be empty of lab jobs.

Step 4. Keep the triage write-up and remove the working directory:

rm -rf deletes the whole lab directory, including anything you added

The next block removes ~/ansible-triage-lab and everything under it, recursively and without confirmation. If you saved notes, an inventory you want to keep, or a playbook you extended, copy it out first — the step above does that for the lab’s own deliverables and knows nothing about yours.

The path is written in full and anchored to $HOME deliberately. A relative rm -rf run from the wrong directory is how people delete work they meant to keep, and the shell gives no second chance.

Confirm what you are about to delete before you delete it:

ls -la "$HOME/ansible-triage-lab"

mkdir -p "$HOME/ansible-lab-deliverables/triage"
cp -a reports/triage.md establish-truth.yml \
      "$HOME/ansible-lab-deliverables/triage/"

rm -rf "$HOME/ansible-triage-lab"

Under the hood

unreachable and failed come from different layers and that is why they behave differently.

failed is a module result: the module ran on the target, decided it could not do its job, and returned a structure with failed: true. The connection worked. Ansible knows exactly what happened and the host stays in the play for handler purposes, though it is removed from subsequent tasks.

unreachable is a connection-plugin error: the transport could not establish or maintain a session. No module ran. Ansible knows nothing about the host’s state beyond “I could not talk to it”, and by default the host is removed from the play entirely for every remaining task — which is why a diagnostic play without ignore_unreachable produces no information about precisely the hosts you care about.

The recap counters are cumulative per host across the play, which is what makes ok=1 changed=1 unreachable=1 expressible at all: the host completed one task and then the connection died. A host that is unreachable from the first task shows ok=0 changed=0 unreachable=1, and the difference between those two lines is the difference between “unknown partial state” and “untouched”.

Neither counter distinguishes a host that was never scheduled. Such a host has no entry in the stats callback at all, which is why the comm against --list-hosts is the only way to enumerate the fourth class.

What You Learned

unreachable and failed are produced by different layers and mean opposite things operationally. A failed host usually changed nothing; an unreachable host may have changed something and then gone silent.
ok=1 changed=1 unreachable=1 is the line to read first. It is a host in a partial state, and it is the only class you cannot reason about without going and looking.
The recap expresses three classes and there are four. Never-attempted hosts have no recap entry, and comm against --list-hosts is the only way to list them.
ignore_unreachable: true belongs on diagnostic plays and nowhere else. It is what lets a triage play report on the hosts it cannot reach.
class-complete.txt is a claim. You verified all ten hosts from the hosts themselves, including the ones the recap said were fine.
Four responses, and retry is not the default. Retry the clean failures, investigate the simultaneous ones, quarantine what you cannot make known, and roll back only when a mixed state is worse than no change.
Install the undo before the block. The at job that removed the firewall rule was scheduled before the rule existed, which is the only ordering that is safe over a network.

Lab: Partial fleet failure — diagnose, classify, decide

Objectives

Prerequisites