Reporting outward without making it mandatory · Ansible

Everything so far in this part stays on the controller: a log file, a structured artefact, some queries over them. That is deliberate. It is the baseline this course recommends, it costs nothing to operate, and it answers most of the questions an operator actually has.

This lesson is about the things you can add on top, and it is written as an evaluation rather than a recommendation, because each of them buys something real and charges for it in a way that is easy to miss until the charge arrives.

One question decides most of them:

What does this integration answer that the run artefact does not?

If the answer is “nothing, but it is more visible”, that is a legitimate reason — visibility is worth something. It is just a different reason from “we cannot currently answer X”, and confusing the two is how estates end up with three reporting systems and no evidence chain.

The three categories

Notification callbacks push events outward as they happen. The community.general collection carries the familiar ones — community.general.slack, community.general.mail and community.general.log_plays among them — and all of them are collection content, not ansible-core, so they are a pinned dependency like any other.

They answer: did the run happen, and how did it end? — to people who are not watching a terminal.

External run recorders capture the whole run into a queryable store. ARA is the established one: it records plays, tasks, hosts and results into a database and gives you a web interface and an API over them.

It answers: what happened across many runs, without writing queries against files?

Platform job history — AWX or Automation Platform — makes the run a first-class object with an owner, a status, a log and an inventory snapshot. Part XXXIX’s platform object model lesson owns this subject; here it appears only as one of three ways to get run history.

It answers: who ran what, with which credentials, against which inventory — with identity established before the run rather than inferred afterwards.

The costs, stated honestly

Callbacks run in the controller process

This is the cost people meet first and attribute to something else.

A notification callback is code executing inside ansible-playbook, on the controller, synchronously with the run. A webhook with a five-second connect timeout, fired per task, on a play with 200 tasks, adds time to every run — and the failure mode when the endpoint is slow rather than down is worse, because the run does not fail, it just takes twice as long and nobody knows why.

The rule that follows: notify at run level, not at task level. A single message at the end of the run, containing the recap, costs one request. A message per task costs 200, and nobody reads them.

A reporting failure must not be a change failure

The design principle for anything in this lesson:

A production change must not depend on a chat service, a ticketing system or a recording database being available.

The reasoning is straightforward once stated. The runs you most need to complete are the ones during an incident, and an incident is exactly when your chat provider, your ticketing system or your internal database is most likely to be degraded — sometimes by the same underlying cause.

Practically:

Test the integration’s failure path, deliberately, before relying on it. Point it at an unreachable endpoint and run a play. If the run fails, or hangs for minutes, you have created a dependency and you found out cheaply.
Keep timeouts short on anything a callback contacts.
Prefer aggregating at the end over streaming during, so a failed notification costs you a message rather than a run.

Sending run data outward is sending module results outward

Everything from lesson 3 applies, with the boundary moved.

A notification callback that includes task results is transmitting whatever those tasks returned — including anything no_log did not cover — to a third-party service, where it is retained under somebody else’s policy, searchable by everyone in the channel, and subject to their breach history rather than yours.

The chat channel is not a log destination

Two properties of chat make it a bad place for run detail, and both are easy to overlook because the integration is so easy to enable.

It is broadly readable. The set of people in an engineering channel is much larger than the set of people who can log in to the controller, and it grows without anybody re-evaluating the decision.

It is retained and searchable. A rendered configuration file posted into a channel during a failure is there for years, indexed, and exported to whatever archive the organisation runs.

Send status outward: which playbook, which environment, the recap counters, the run id, a link to where the detail lives. Keep content on the controller. If somebody needs the detail, they need the artefact, and the message should tell them where it is.

An external recorder is a system you now operate

ARA and its equivalents give you a real query interface, and in exchange you are running a database and a web service.

That has consequences that are ordinary and easy to underestimate:

It needs backup, patching, capacity and a restore procedure — and it is now part of your audit trail, so losing it loses evidence.
It has an access-control question of its own: it holds task results from every run, which makes it approximately as sensitive as the controller.
It is one more thing that can be down during the incident when you want to query it, which argues for keeping the file artefacts as well rather than instead.

None of that is an argument against it. It is an argument for deciding deliberately, because a recorder adopted casually becomes an unowned service holding your audit evidence.

Platform job history does not cover the runs that bypass it

The strongest of the three, with one specific gap.

A platform records the runs made through the platform. Runs made by somebody with a checkout and a terminal are invisible to it — and adopting a platform tends to create a confident belief that all runs are recorded, which is more dangerous than knowing they are not.

Closing that gap is an access decision rather than a tooling one: the automation account’s credentials live only on the platform, so a laptop cannot reach the fleet at all. Until that is true, the platform’s job history is a record of the disciplined runs and not of the interesting ones.

Choosing

Read-only / Safethe baseline, before adding anything— Prints the effective settings that differ from the defaults, so you can see what the controller is actually configured to record.

ansible-config dump --only-changed

If log_path is set, a structured callback is enabled and artefacts carry the metadata from lesson 4, you already have an evidence chain. Add outward integrations against a named question:

Question	Smallest thing that answers it
Did the nightly run happen and how did it end?	A run-level notification, or an alert on artefact absence
Which host did this run touch?	The artefact — no integration needed
What changed across the fleet last month?	Queries over retained artefacts
Who ran this, established rather than inferred?	Platform job history, or CI with authenticated requesters
Can a non-operator see run history without shell access?	An external recorder, or a platform
Is this run slower than usual?	Timing callbacks plus retained artefacts

Two rows in that table are answered by things you already have. That is the point of the table.

Under the hood

Why does an exception inside a callback have a different blast radius from an exception inside a module?

A module runs in its own process — usually on the managed node — and its failure is a result: the task fails, the host drops out, the play continues for everyone else. Ansible’s whole error model is built around that.

A callback runs inside the controller process, invoked directly by the task engine as part of dispatching events. It is not a task, it has no result object, and there is no host for its failure to be attributed to. Ansible does guard against a badly behaved callback taking the run down, but the guarding is not something to rely on for a plugin you configured yourself: a callback that blocks rather than raises — a socket waiting on a connect timeout — is not an exception at all, and nothing will interrupt it.

Three practical consequences:

Blocking is worse than failing. A notification endpoint that refuses connections fails fast. One that accepts the connection and never answers stalls the run for the full timeout, per event.

Callback configuration is global. Enabling a notification callback in ansible.cfg applies it to every playbook in the repository, including the read-only ones and the --check runs. Most teams want notifications for production changes and not for a dry run, and the way to get that is to enable the callback through the environment in the pipeline that runs production changes, rather than globally in the repository.

Order is not guaranteed to help you. All enabled callbacks receive the same events; there is no dependency between them. A notification callback does not wait for the artefact to be written, so a message that links to an artefact may arrive before the file exists. Link to a location, not to a promise that the file is there yet.

What happens if the chat integration is down during an incident?

Two very different outcomes, and which one you get was decided months earlier when somebody enabled the callback.

If notification is on the critical path — a callback with a long timeout firing per task — the run slows to a crawl or fails outright. The change that would have fixed the incident does not go out, and the reason is that a chat provider is having its own bad day. In the worst version the two are the same bad day, because the outage that took out your service also took out the network path to the provider.

If notification is advisory — one aggregated message at the end, short timeout, failure ignored — the run completes normally and nobody is told about it in chat. Somebody has to look at the artefact instead, which is mildly annoying and entirely survivable.

Getting the second outcome takes three decisions:

Aggregate at the end of the run, so there is one request rather than hundreds.
Short timeouts, so a hung endpoint costs seconds rather than minutes.
Test it broken. Point the integration at an unreachable address and run a play. Measure how much longer the run took and whether it still succeeded. This is a ten-minute experiment that tells you what your automation does on the worst day, and almost nobody runs it.

The same three apply to a ticketing integration, a metrics push, and any uri task in a play that reports outward. The general principle is one this course keeps returning to: the reporting must never be able to prevent the change.

Adding an outward integration

Name the question it answers that the artefact does not. If there isn’t one, the honest reason is visibility, which is fine — say so.
Check whether it is collection content. Every notification callback is; pin it in requirements.yml.
Aggregate at run level. One message per run, not one per task.
Send status, not content. Playbook, environment, recap counters, run id, and a pointer to where the detail lives.
Set a short timeout and test the failure path against an unreachable endpoint before you rely on it.
Enable it where it belongs — in the pipeline that runs production changes, not globally in ansible.cfg where it also fires on every dry run.
If it is a recorder, give it an owner, a backup and a restore test. It now holds audit evidence.
If it is a platform, close the bypass. Job history covers the runs made through it and nothing else.

Knowledge check

Why this matters

The temptation with observability is to buy visibility and call it evidence. A chat channel that announces every deployment feels like a record. It is not one — it is a feed, it is readable by more people than should read it, and it will be gone or unsearchable exactly when somebody asks a precise question about a change from four months ago.

The evidence is the artefact. It sits on the controller, costs nothing to operate, contains the per-task detail, and answers the questions in lessons 5 and 6. Everything in this lesson is a layer on top of that, and each layer should be justified by a question the artefact cannot answer.

The other half of the argument is the one that recurs through the whole course: a control that can stop a change is a control that will stop a change at the worst moment. Reporting exists to tell people what happened. The instant it can prevent something from happening, it has stopped being observability and become a dependency of production — usually without anyone deciding that it should be.

Test it broken. It takes ten minutes and it is the only way to find out which of the two you have.