Metrics that predict an automation failure · Ansible

Automation does not usually break suddenly. It rots, and the rot is measurable for weeks before it produces an incident.

The four measurements in this lesson are chosen on one criterion: each of them moves before the failure, not during it. A metric that goes red when the run fails has told you what the run already told you. These tell you that the run will fail, or that the fleet has stopped being what your automation assumes.

They also share a weakness worth stating first. All four are computed from the run artefacts of lesson 4, and none of them is meaningful if the underlying changed, timing and host data is dishonest. Every section therefore ends with the practice that destroys it.

1. Drift rate

Definition. The number of tasks reporting changed per run, on a fleet that has already converged, excluding runs that deployed something.

Healthy baseline: zero. A converged fleet running the same automation against unchanged inventory should change nothing. That is what convergence means, and it makes drift rate the rare operational metric with an unambiguous target rather than a threshold somebody argued about.

What a rise means. Something outside Ansible changed the hosts and this run corrected it. That is the definition of drift, and the causes are worth enumerating because they are diagnosable:

Someone made a manual change. The snowflake servers lesson is about what happens next.
A package upgrade replaced a configuration file that your automation then re-templated.
A second tool owns the same state, and the two are alternating — the failure from the ownership lesson in Part XLII. This one shows up as a drift rate that is constant rather than spiky, on the same task, on the same hosts, forever.
The resource genuinely changed for a legitimate reason nobody recorded, which is what intentional exceptions exists to make visible.

What destroys it. A single task with an inaccurate changed_when. If one task reports changed on every run, the fleet’s drift rate never returns to zero, so there is no baseline to deviate from. Within a couple of weeks nobody reads the number, and the estate has quietly stopped being audited. The tasks that always change lesson treats this as an anti-pattern for handler reasons; this is the second, larger cost.

The corollary is a rule worth adopting: a task that always reports changed is a defect with a deadline, not a cosmetic issue.

2. Run-duration distribution

Definition. Wall-clock time per run, tracked as a distribution across runs — not a mean.

Why the distribution. The mean hides the two shapes that matter. A run whose median is stable but whose 95th percentile has doubled has a subset of hosts getting slow, which is a fleet problem. A run whose whole distribution has shifted has an automation problem. Those need different investigations and a single average number reports them identically.

Why it predicts failure. Duration is the metric that turns into an operational constraint rather than an alert. A convergence run that has grown from twelve minutes to fifty has not broken anything — until the day it has to run inside a thirty-minute change window, or during an incident when somebody needs a configuration pushed now. At that point the growth you did not track becomes the reason the fix is late.

Duration also aggregates several underlying causes that are worth separating once you notice it:

Fact gathering, which the fact gathering and its cost lesson quantifies, and which grows with the fleet.
Connection setup, addressed by pipelining and connection reuse.
A task that has become slow — a package repository that got further away, a command that now waits on something.
More hosts, which is the benign explanation and should be checked first by normalising to time-per-host.

How to localise it. Total duration says a run got slower; per-task timing says which task. ansible.posix.profile_tasks and ansible.posix.timer are the callbacks for this, enabled as lesson 2 describes and pinned as collection dependencies.

What destroys it. Comparing runs that are not comparable. A run with --limit against forty hosts and a full run against four hundred are different measurements, and mixing them produces a duration series that is noise. Normalise by host count, and record the limit in the artefact — which lesson 4 already requires for other reasons.

3. Unreachable-host trend

Definition. The count of hosts reporting unreachable, per run, and — more usefully — the number of consecutive runs each host has been unreachable for.

The second form is the one that carries information. A single unreachable host on one run is a host that was rebooting. The same host unreachable for fourteen consecutive nightly runs is one of:

A machine that was decommissioned and never removed from inventory. This is the common case, and it is a slow leak: your inventory is gradually diverging from reality, and the divergence is invisible in every individual run because one unreachable host looks like noise.
A host that has fallen off the network and nobody noticed, because the only thing that talks to it is the automation.
A credential or key that stopped working on that host, which will eventually stop working on all of them.

The trend matters more than the count precisely because the count is tolerable at every point on the way up. Three unreachable hosts out of four hundred is fine. It stays fine at five, at eight, at twenty — and at some point you have an inventory that nobody trusts, which the inventory as the blast radius map lesson identifies as the precondition for every safety control in this course.

What destroys it. Two things.

Building preflight checks from debug. As lesson 1 showed, action plugins that run on the controller never connect, so a reachability play built from debug reports unreachable=0 for hosts that are comprehensively unreachable. Measure with something that connects, such as ansible.builtin.ping.

And forgetting that unreached hosts are absent from the recap rather than zero. A host that was targeted but never reached because the play stopped early is not counted as unreachable — it is not counted at all. The unreachable metric therefore needs the intended target list as its denominator, or it silently under-reports exactly when a run went badly.

4. Per-task failure hotspots

Definition. For each task, across runs, the proportion of host executions that failed — including failures that were ignored or rescued.

Why including the suppressed ones is the point. A task with ignore_errors: true that fails on 3% of hosts every single run does not appear in failed at all. It appears in ignored, which nobody trends, and the run is green. That is a real defect running continuously with a suppression on top of it, and the suppression was almost certainly added during an incident by someone who intended to come back to it.

The same is true of rescued. A rescue block that fires routinely means the primary path routinely does not work — the automation is running on its error path as normal operation, and that is a design statement nobody made deliberately.

Why it predicts failure. A task failing on 3% of hosts is a task that will fail on all of them when the underlying condition becomes universal. A package that is missing on three old hosts is a package that will be missing everywhere after the next distribution upgrade. The 3% is a sample of the future.

What destroys it. Aggregating at run level instead of task level. “The run succeeded” is true for every one of these cases. The hotspot only appears when you count per task, per host, across runs — which is exactly why the artefact has to keep per-task results rather than the recap summary.

Computing them

All four come out of the run artefacts. With the JSONL callback from ansible.posix writing one JSON object per event, the shape of the computation is:

Read-only / Safethe shape of the drift-rate and hotspot queries— Reads artefact files and prints counts. Adjust the field paths to the callback you have enabled - artefact schemas differ between callbacks and versions.

# Changed tasks per run, most recent 30 runs
for f in $(ls -1t artefacts/*.jsonl | head -30); do
printf '%s %s\n' "$f" "$(jq -r 'select(.changed == true) | .task' "$f" | wc -l)"
done

# Tasks by how often they report changed, across all retained runs
jq -r 'select(.changed == true) | .task' artefacts/*.jsonl \
| sort | uniq -c | sort -rn | head -20

Two cautions about that snippet, and they are the reason it is written as a shape rather than a recipe.

Artefact schemas differ. The field names emitted by a JSON callback are a property of that callback and its version. Read one artefact by hand before writing anything that parses a hundred of them.

The second query is the highest-value thing in this lesson. Tasks ranked by how often they report changed, across all retained runs, on a converged fleet. The top of that list is either your genuine drift, or your lying tasks, and either way it is a short list of specific things to fix.

What to alert on, and what to review

These are not alerts, with two exceptions. Trends are for review; pages are for outages.

Alert on: a run that did nothing (ok=0 with non-zero skipped, from lesson 1), and a run that did not happen at all — the scheduled job that silently stopped is invisible in every metric here, because metrics computed from artefacts cannot see a missing artefact.

Review, on a schedule somebody owns: drift rate against a target of zero; duration at the median and the 95th percentile, normalised per host; hosts unreachable for more than N consecutive runs; and the top ten tasks by failure or change frequency.

A monthly half hour with those four is worth more than a dashboard nobody opens, and it produces a specific list of defects rather than a feeling that things are fine.

Under the hood

Why is changed the load-bearing signal, when it is entirely self-reported?

Because it is the only place in the system where a module states an intention about the world rather than about itself. ok says the module ran. failed says it did not. changed says the module altered something that was not already correct — which is the one claim that can be compared against an expectation.

That is also its fragility. The callback counting changed never saw the host; it is counting result.changed from the module’s return value and performing no verification, because it is structurally incapable of any. A command task with changed_when: true increments the fleet drift metric with no change having occurred anywhere, and nothing downstream can tell.

Which is why the accuracy of changed_when is a fleet-level property rather than a per-task style question. Every well-written module derives changed from an actual comparison — read the current state, compare, act. Every command and shell task delegates that judgement to whoever wrote the task, and by default a command reports changed unconditionally because it has no way to know.

Two habits follow, and they are the whole of drift hygiene:

changed_when: false on anything that only reads. A show command, a status query, a lookup. It costs one line and it protects the metric.
A real predicate on anything that acts. Derive changed from the command’s own output — a return code, a marker in stdout — as the changed_when as a precision tool lesson sets out.

The fleet’s ability to detect drift is the sum of these decisions, made one task at a time, mostly by people who were not thinking about metrics.

What happens if the drift rate has never been zero?

You cannot use the metric until you fix that, and the fix is a specific, finite piece of work rather than a programme.

Run the ranking query: tasks ordered by how often they report changed across retained runs. On a converged fleet, that list is your problem inventory, and it is usually far shorter than people expect — a handful of tasks account for nearly all of it.

Then triage each one into three buckets:

Lying tasks. A command with no changed_when, a shell that always reports change, a task that rewrites a file with a timestamp in it. Fix with changed_when: false or a real predicate. This is most of the list and it is a day’s work.
Genuine repeated drift. Something outside Ansible really is changing the host every day. Find it. It is usually a package post-install script, another configuration system, or a human with a habit.
Two owners. The task and something else both manage the same state and they alternate. This is the Part XLII ownership problem; resolve it by deleting one of the two declarations, not by making both smarter.

When the list is empty the baseline is zero, and every future non-zero reading is a question with an answer. Until then, every reading is noise and the fleet is not being audited — regardless of how much automation is running, or how green it is.

The four metrics, and the sloppiness that voids each

Metric	Healthy	Voided by
Drift rate	Zero on a converged fleet	One task with an inaccurate `changed_when`
Run duration	Stable median and p95, per host	Mixing limited and full runs in one series
Unreachable trend	Zero, or a known short list	Preflight checks built from `debug`; ignoring that unreached hosts are absent rather than zero
Per-task failure rate	Zero, including ignored and rescued	Aggregating at run level, where everything looks successful

And one thing none of them can see: the run that never happened. Alert on artefact absence separately, because a scheduled job that silently stops produces perfect metrics forever.

Knowledge check

Why this matters

Most estates measure whether the last run succeeded. That is a retrospective, binary fact about an event that is already over, and it answers a question nobody urgently needs answered — the failure was loud.

The four measurements here answer a different question: is this automation still fit for the fleet it is aimed at? And they answer it early. Drift rate rises weeks before somebody discovers a hand-edited server. Duration grows for months before it collides with a change window. The unreachable trend climbs quietly while the inventory becomes fiction. A task fails on 3% of hosts for a year before the condition becomes universal.

None of that requires a monitoring platform. It requires artefacts with per-task detail, an honest changed, and half an hour a month with four queries.

The honest changed is the part that is hard, because it is the sum of hundreds of small decisions made by people who were solving a different problem. That is why this part keeps returning to it: a single careless command task does not just produce a noisy line in a run — it removes the fleet’s ability to be audited, and nothing ever reports that it did.