Compliance on a schedule, read by someone · Ansible

The last lesson of this part is about the failure it cares about most, and it is not a technical one.

An audit job nobody reads is worse than no audit, because it manufactures the belief that someone is looking.

An estate with no compliance automation has a known gap. Somebody may eventually address it. An estate with a nightly compliance run producing a report that nobody has opened in seven months has a gap that is invisible, is documented as covered, and appears in the control register with a green tick.

The second one is worse, and everything in this lesson is about not building it.

Running the audit on a timer

Mechanically simple. The audit playbook cannot enforce — that separation is from lesson 1 — so scheduling it needs no change authorisation.

Read-only / Safethe scheduled invocation— --check --diff cannot change a target. The run ID ties the evidence files to this invocation.

#!/bin/sh
set -eu
RUN_ID="$(date -u +%Y%m%dT%H%M%SZ)"

ansible-playbook -i inventories/prod compliance-audit.yml \
--check --diff \
-e "compliance_run_id=${RUN_ID}" \
-e "compliance_policy_version=$(git -C . rev-parse --short HEAD)"

rc=$?
printf 'run=%s rc=%s\n' "$RUN_ID" "$rc" >> /var/log/compliance/runs.log
exit "$rc"

A systemd timer is generally better than cron for this: it records the last run in a way that is queryable, handles a missed run explicitly through Persistent=true, and puts the job output in the journal without a mail alias. cron works, and the ansible.builtin.cron module manages it if that is the estate convention.

The scheduling detail that matters more than either: stagger the run across the fleet or accept the load spike. 3,000 hosts audited at 03:00 is 3,000 SSH connections and 3,000 fact gatherings in a window, which is a self-inflicted load event with a compliance justification.

The exit code is not the finding

ansible-playbook --check exits 0 for a run in which 47 hosts are non-compliant. Predicted changes are not failures.

A scheduled job that alerts on non-zero exit therefore alerts on broken audits, not on drift — which is worth having, and is not the thing anybody thinks they configured. The two signals are different and both are needed:

Signal	Source	Means
Non-zero exit	`ansible-playbook`	The audit failed to run properly
Exit 4	Unreachable hosts	Part of the estate was not measured
`changed` count	The recap	Hosts differ from the baseline
Findings in the evidence files	The run’s artefacts	What differs, and where

A monitoring setup that only watches the exit code has an audit that reports success while the estate drifts, and it will do so indefinitely.

Alert on the rate, not on the diffs

An estate of 300 hosts has drift. Some of it is being worked on, some is excepted, and some is genuinely new. Alerting on every finding produces a message per finding per run, which produces a filter rule, which produces silence.

The useful signals are derived:

Signal	Why it is the right thing to alert on
Newly failing controls	A host that passed yesterday and fails today is an event with a cause
Drift rate	Findings per day, trending. A rise means a process changed
Hosts newly non-compliant	Points at a host, which points at a change
Controls with rising failure counts	Points at a control, which may be wrong
Hosts unmeasured across N runs	The most serious finding, and the quietest
Exceptions past their review date	The mechanism by which exceptions become permanent

None of those is “host web41 has PermitRootLogin yes”. That is a finding, it belongs in the report, and it is not an alert — because it was also true yesterday, and the day before, and alerting on it every night is how the channel becomes noise.

Tuning out noise without hiding it

Some findings are expected. A build host that legitimately has compilers installed, an appliance whose vendor requires a setting the baseline forbids, a group mid-migration.

Two ways to make those stop generating alerts, and only one is safe.

The wrong one: remove the control for those hosts, or add a when that skips them. The finding disappears from the report. Six months later nobody knows the control is not being evaluated there, and the report claims coverage it does not have.

The right one: the exception mechanism from lesson 1 — recorded in inventory, with a reason, a compensating control, an approver and a review date. The control still runs and still produces its finding; the finding is classified as excepted rather than non-compliant.

Read-only / Safethree categories, not two— The excepted category is what makes a compliance percentage believable. A report showing 100% on an estate everyone knows has exceptions is a report nobody trusts.

- name: Classify this finding
ansible.builtin.set_fact:
  finding_status: >-
    {{
      'compliant' if control_passed
      else ('excepted' if control_name in (compliance_exceptions | default({}))
            else 'non_compliant')
    }}

- name: An exception past its review date is a finding in its own right
ansible.builtin.assert:
  that:
    - compliance_exceptions[control_name].review_date is version(
        ansible_date_time.date, '>=')
  fail_msg: >-
    Exception for {{ control_name }} on {{ inventory_hostname }}
    expired on {{ compliance_exceptions[control_name].review_date }}
  quiet: true
when: finding_status == 'excepted'

The second task is what stops exceptions becoming permanent. An exception with no expiry is a decision never to apply the control, written in a way that avoids anyone having to say so.

How a compliance job stops being read, in five predictable stages

Worth laying out, because recognising the stage you are at is most of the remedy.

Stage 1 — it is new and interesting. The first runs find real things. People read them. Controls get fixed. The programme demonstrably works.

Stage 2 — the remaining findings are the hard ones. What is left is the legacy appliance, the vendor requirement, the host mid-migration. Each run reports the same twelve findings. The report is still read, but skimmed, because the reader knows what is in it.

Stage 3 — the report becomes a diff nobody computes. The only useful information is what changed since yesterday, and the report does not say — it lists everything, every time. Extracting the change means comparing two reports by hand, which nobody does.

Stage 4 — the alert is filtered. It fires nightly with the same content, so somebody writes a rule. Nothing is lost immediately, because the content genuinely was the same. The rule outlives the reason.

Stage 5 — the job breaks and nobody notices. An inventory change, a credential expiry, a Python upgrade on the controller. The job exits non-zero into a filtered channel. The dashboard still shows the last successful assessment, which ages quietly. The control register says compliance is monitored continuously.

The transition that does the damage is stage 3, and it is a design defect rather than a discipline failure: the artefact does not answer the question its reader has. Two things prevent it, both cheap if built at the start:

The run emits a delta, computed against the previous run’s evidence directory. New findings, resolved findings, unchanged count. That is the artefact somebody reads daily.
The freshness of the assessment is itself monitored. An alert when the newest evidence directory is older than the schedule allows — which is the only thing that catches stage 5, and it is monitoring the monitor, which is exactly the right amount of paranoia for a control whose failure mode is silence.

What happens if the audit takes longer than its interval?

Runs overlap, and the failure is untidy rather than dramatic — which is why it persists.

Two runs against the same fleet at once means double the SSH connections, contention on the fact cache, and two sets of evidence files with interleaved timestamps. Under systemd, a service unit will not start a second instance while the first is running, so the timer’s trigger is skipped and the schedule silently degrades to whatever the run duration is. Under cron, the second run starts, and both continue.

Three responses, in order:

Measure the duration and set the interval from it. An audit taking 90 minutes does not run hourly. This sounds obvious and the interval is usually chosen before anyone has timed a full run.
Make the run cheaper. gather_subset scoped to what the policy needs, fact caching between runs, and a strategy suited to the fleet. Part XXXIV covers the mechanics.
Split the fleet. Audit a third of the estate every night rather than the whole estate every three nights. Same total load, every host measured on the same cadence, and each run finishes inside its window.

Whichever applies, the fix belongs alongside a --limit in the scheduled invocation and a lock — a single-instance guard — so that a slow run degrades to a skipped run rather than to two runs fighting. And a skipped run must itself be visible, or you are back to stage 5.

Knowledge check

Why this matters

Every part of this course has argued that a control which cannot fail is not a control. This lesson is the same argument applied to the compliance programme itself.

A nightly audit that runs, produces artefacts, and is never read cannot fail in any way anyone would notice. It produces exactly the same output whether the estate is perfectly compliant, badly drifted, or not being measured at all — because in each case the output is a file nobody opens. The control has one output state, and a control with one output state is decoration.

The test is the same one from the maintenance-window lesson, asked of the programme rather than of a change: what would we have seen this week that would have caused somebody to act?

If the answer is a delta somebody read and responded to, the programme is working. If the answer is that the dashboard is green, ask when it last was not — and if nobody can remember, that is the finding.