Three hundred hosts, two hundred and twenty successes

It is 02:47. A scheduled patch and configuration run against 300 production hosts has finished. This is the last thing it printed:

$ tail -n 6 /var/log/ansible/patch-2026-08-11.log

PLAY RECAP *********************************************************************
220 hosts : ok=31   changed=27   unreachable=0   failed=0
 30 hosts : ok=18   changed=14   unreachable=0   failed=1
 50 hosts : ok=0    changed=0    unreachable=1   failed=0

real    41m12s
exit=4

Someone asks the question that this entire course exists to make answerable: was the change successful?

The honest answer is that the question has no answer in that form, and producing one anyway is what turns a bad night into an incident. What you can answer, and must, is a different set of questions.

Why “73% success” is the wrong summary

220 of 300 is 73.3%. That number is arithmetically correct and operationally useless, for three reasons.

It averages populations that need different work. Thirty hosts that ran a module and got a negative result and fifty hosts that never answered the phone are not 80 units of the same thing.

It implies a threshold that does not exist. There is no percentage at which a partial configuration change becomes acceptable. Whether 73% is fine or catastrophic depends entirely on which 73% and what the change was. Seventy-three percent of a monitoring agent rollout is progress. Seventy-three percent of a TLS certificate rotation is an outage with a scheduled start time.

It hides the real result. The real result is not a percentage. It is this:

That is the sentence to say in the incident channel. Not “73% worked”.

The classification, in the order it must be done

Do not start by fixing anything. Start by producing four lists of hostnames. Every subsequent decision needs them, and they take minutes to produce from the run artefact.

1. The unreachable list — before it changes

log=/var/log/ansible/patch-2026-08-11.log

# Hosts that never answered: unreachable=1 and ok=0
awk '/unreachable=1/ && /ok=0/ {print $1}' "$log" | sort -u > unreachable.txt

# Hosts that ran something and then failed
awk '/failed=[1-9]/ {print $1}' "$log" | sort -u > failed.txt

# Hosts that dropped part-way: unreachable but with completed tasks
awk '/unreachable=1/ && !/ok=0/ {print $1}' "$log" | sort -u > dropped-midrun.txt

# Everything else in the target group
comm -23 targeted.txt <(cat unreachable.txt failed.txt dropped-midrun.txt | sort -u) \
> succeeded.txt

wc -l unreachable.txt failed.txt dropped-midrun.txt succeeded.txt

The third list is the one people forget, and it is the reason the extraction is worth scripting rather than eyeballing. A host with unreachable=1 ok=0 never started. A host with unreachable=1 ok=17 ran seventeen tasks and then vanished. Those belong in different buckets: the first is stale, the second is partially converged and has an in-flight task whose outcome nobody knows.

2. Sub-classify the 30 failures by where they stopped

Thirty hosts that failed the same task are one problem with one fix. Thirty hosts that failed thirty different tasks are thirty problems, and the run is telling you the fleet is not homogeneous.

grep -E '^(fatal|failed):' "$log" \
| sed -E 's/^[a-z]+: \[([^]]+)\].*/\1/' \
| sort | uniq -c | sort -rn | head

# and the task each host was on, if the callback recorded task names
grep -B1 -E '^fatal: \[' "$log" | grep '^TASK' | sort | uniq -c | sort -rn

3. Sub-classify the 50 unreachables by error text

The SSH error text is a taxonomy, covered in full in the transport part. For triage, three buckets are enough: connection refused or timed out (host down, or network path broken), permission denied (the host is up and your credentials are the problem), and host key mismatch (the host is up and is not the host you expected). The three have completely different implications, and the third is a security event until proven otherwise.

4. Now, and only now, check the 220

This is the step almost everyone skips, and it is the one the scenario is really testing.

Are the 220 consistent with each other?

changed=27 on one host and changed=27 on another does not prove they did the same 27 things. Two hosts can report identical counters having converged to different configurations, because the play contains conditionals, because a template rendered different values from different group vars, or because a package resolved to a different version from a different mirror.

Ansible reports what its modules claimed, per task, per host. It does not compare hosts to each other. Nothing in a recap tells you that the fleet is uniform, and the recap is the artefact people treat as proof of uniformity.

$ ansible -i inventory/prod --limit @succeeded.txt appservers -m command -a 'sha256sum /etc/app/app.conf' -o | awk '{print $NF}' | sort | uniq -c

    197 4f1c...a92e  /etc/app/app.conf
   21 8b30...11d7  /etc/app/app.conf
    2 e77a...5c04  /etc/app/app.conf

Three distinct hashes across a population the recap called uniformly successful. That is the finding that changes the decision — and it was invisible in the run output.

The four questions, and what evidence each needs

Those four are the shape of the next five lessons. The point of stating them here is that they are answerable in a defined order, and the order matters: you cannot judge rerun safety before you know which hosts you would be rerunning against.

What to say, and to whom

Triage is not finished when you understand the situation; it is finished when the people who need to act know it. The statement has a shape:

• What ran, against what, and when. “Patch and config run, 300 production app servers, started 02:06, ended 02:47.”
• The three populations, with counts and names available. Not a percentage.
• What is now unsafe to assume. “Any procedure that assumes all app servers are on the new config is unsafe until reconciliation.”
• What you are doing next, and what decision you need from whom.

That last line is the one that matters at 03:00. The four-way decision — retry, roll back, quarantine, investigate — is often not yours alone to make, and the criteria for it should have been written into the change plan before the run started. Lesson 7 of this part is about writing them down in advance, because nobody makes that judgement well while a half-converged fleet is on fire.

Knowledge check