Detect, decide, enforce

Detection produces a list. What happens next is where drift programmes succeed or quietly stop existing.

The failure is predictable: the report arrives, somebody looks at it, somebody runs the playbook for real against everything to “fix it”, a service restarts during business hours, and the drift report is quietly descoped to a weekly email nobody opens.

The structure that avoids that has three stages, and the middle one is a human.

Stage	What runs	Blast radius	Who decides
Detect	The roles, `--check --diff`, whole scope, scheduled	None (see the caveats in lesson 3)	Nobody — it is a report
Decide	A person reading the report	None	Named owner
Enforce	The roles, for real, against a chosen scope	Real, and stated in advance	Change process

The stage that gets skipped is decide, because when the fix is one command away it feels like a formality. It is not: a drift finding has four possible correct outcomes and only one of them is “run the playbook”.

Triage: four outcomes, one of which is enforcement

For each finding:

1. Real drift — enforce. The host changed, the declaration is right, the host should be corrected. Run the role against it, in waves.

2. An undeclared requirement — change the code. The host is right and the declaration is wrong. Somebody made a deliberate improvement out of band and it should become the standard. Enforcing here would undo a correct change, which is the outcome that makes people distrust automation.

3. A legitimate deviation — record an exception. This host genuinely must differ: a licensing constraint, a customer requirement, a device that will not tolerate the standard setting. Enforcing breaks it; doing nothing means it appears in every report forever. It needs the mechanism in the next lesson.

4. A reporting defect — fix the task. The host and the declaration agree and the task claims otherwise: a rendered timestamp, an unfenced command, a mode expressed as a string. Enforcing changes nothing real and the finding returns tomorrow.

Separating audit from enforcement without duplicating anything

The instruction “keep the audit playbook separate from the enforcement playbook” is right and is usually implemented wrongly, by writing a second playbook full of command and assert tasks that check the same things the roles declare.

That duplicate immediately starts drifting from the roles. Within months the audit passes on hosts the enforcement would change, which is the worst possible failure for a compliance control.

Separate the entry point, not the logic. One set of roles, two thin playbooks:

Read-only / Safeaudit.yml— The audit entry point. Identical role list to the enforcement playbook; the difference is that handlers are inert here because check mode makes no change, and the playbook is only ever invoked with --check.

- name: audit the web tier against its declaration
hosts: "{{ target | default('env_prod') }}"
gather_facts: true
roles:
  - common
  - nginx
  - monitoring_agent

Service impact possiblesite.yml— The enforcement entry point. Same roles. This one changes hosts and restarts services, which is why it is a different file with different approvals attached to it.

- name: converge the web tier
hosts: "{{ target | mandatory }}"
gather_facts: true
roles:
  - common
  - nginx
  - monitoring_agent

Two details in there are deliberate.

target | default('env_prod') in the audit: a scheduled report should cover everything by default, and a missing variable should not produce a zero-host run.

target | mandatory in the enforcement playbook: a missing variable must be an error. An enforcement run with an unset hosts pattern that silently falls back to all is the single most expensive typo in this course.

Enforcing in waves

Remediation is a fleet-wide change and gets the same treatment as any other: waves, gates, a stated blast radius. Converging four hundred drifted hosts in one run is exactly the operation the wave discipline exists to prevent.

Service impact possibleremediate one finding, one wave at a time— Tags restrict the run to the role that produced the finding; --limit restricts it to the wave. Both narrow the blast radius, and --list-hosts first tells you the number before you commit to it.

# what would this touch?
ansible-playbook -i inventory/ site.yml \
-e target=wave1_canary --tags nginx --list-hosts

# see the change before making it
ansible-playbook -i inventory/ site.yml \
-e target=wave1_canary --tags nginx --check --diff

# then, deliberately
ansible-playbook -i inventory/ site.yml \
-e target=wave1_canary --tags nginx

Narrowing by tag matters as much as narrowing by host. A finding in the nginx role does not justify running common and monitoring_agent against those hosts as well — that is three roles’ worth of blast radius to fix one file.

Convergence on a schedule: the honest trade

Running the enforcement playbook automatically, on a timer, against the fleet, is called unattended convergence. It is a legitimate and widely used design, and it is a change-management decision rather than a convenience.

What you get:

Drift is corrected within one cycle rather than within one triage meeting. The window during which a host is wrong is bounded.
The estate genuinely stays converged, so changed=0 remains a meaningful signal rather than an aspiration.
Manual out-of-band changes do not persist, which removes the incentive to make them.

What you give up:

Production changes with nobody watching. At 02:00 a handler may restart a service, and the first person to know is whoever is paged.
The content gate becomes the only gate. Whatever is on the main branch at 02:00 is what gets applied to the fleet. A merged mistake reaches production without anyone running it.
Blast radius is the whole declared surface on every host, every night, by default.
The out-of-band change gets reverted mid-incident. Someone applying an emergency fix by hand at 01:30 finds it undone at 02:00, and diagnosing that while the outage is live is genuinely awful.

Preconditions for unattended convergence

If you turn it on, these are not optional:

The play is proven idempotent by a second-run test in CI, on every role, not just believed to be.
Handlers reload rather than restart wherever the service supports it, and any that must restart are known and listed.
It is waved. The scheduled job runs against non-customer-facing hosts first, on a different schedule from customer-facing ones.
There is an off switch that does not require a merge, and everyone on call knows where it is.
Every run is recorded — what changed, on which hosts, from which commit — somewhere an incident responder will find at 02:30.
An emergency-change convention exists so a manual fix is not silently reverted: usually a host-level flag that removes the host from the scheduled scope until someone clears it.

Point 6 is the one that gets discovered the hard way.

The middle position is often the right one: converge the boring parts, report on the risky parts. Monitoring agents, sudoers, SSH configuration and package baselines are converged unattended; the application tier is reported and enforced deliberately. That splits by consequence rather than by tooling, which is the same instinct the wave work applies to hosts.

Under the hood

There is nothing in Ansible that distinguishes an audit run from an enforcement run other than the --check flag and the pattern you supplied. The same roles, the same tasks, the same modules, the same variable resolution.

That is a strength, and it is why the separation belongs at the entry point. The alternative — a separate audit implementation — has to re-derive what “correct” means, and it can only be as accurate as the last time someone synchronised it with the roles. A configuration audit that disagrees with the configuration management is worse than none, because it produces confident false negatives.

The same identity explains why the drift report’s coverage is exactly the enforcement play’s coverage, minus the tasks check mode cannot examine. You cannot audit more than you declare, using this technique, and increasing audit coverage means increasing the declared surface — which increases enforcement blast radius too. They move together.

What happens if a remediation run makes things worse?

Rolling back a remediation is not symmetric with applying it, and it is worth being clear about that before you need it.

The declaration rolls back cleanly. Revert the commit, run the roles again, and the files return to their previous declared content. That part is genuinely reversible and it is the main argument for having everything in the repository.

The side effects do not. A service that was restarted cannot be un-restarted. A package upgraded to satisfy a state: latest will not come back by reverting the playbook, because the playbook never named the old version. Connections dropped during the restart are gone, and any state that lived in memory with them.

So the rollback plan for a remediation has two halves: revert and re-run for the configuration, plus an explicit statement of what the side effects were and whether they are acceptable a second time. If the answer is “restarting the service again is itself an outage”, then the rollback needs its own change window, and knowing that before the remediation is what makes the difference.