Idempotency and change detection — the production discipline

Idempotency is the property that makes Ansible safe to re-run. Change detection is the property that makes a fleet safe to operate at scale. Both are required: a role that is not idempotent is unsafe to re-apply after a partial failure, and a fleet that has no change detection is unsafe to operate because the operator cannot tell what changed and when. This lesson is about both, and about the production discipline that emerges when both are in place.

Idempotency as a property across applies

A role is idempotent if running it twice against a host produces the same end state as running it once. The first apply mutates the host to the desired state; the second apply reports zero changes. The property is across applies: it is meaningless to talk about idempotency for a single apply, because every apply mutates the host at least once.

flowchart LR
    A["First apply"] --> B["Host converged to desired state"]
    B --> C["Second apply"]
    C -->|"role is idempotent"| D["Zero changes reported"]
    C -->|"role is non-idempotent"| E["Host mutated again"]

The diagram makes the operational consequence obvious. A non-idempotent role is a role that mutates state on every run. If the first apply succeeds, the second apply may reinstall packages, rewrite configuration files, restart services, or trigger handlers that have already run. A partial failure between the first and second applies leaves the host in an unknown state that the second apply will then drift further.

Three classes of mistake produce non-idempotent roles:

• Shell and command without creates or removes. A task like command: /usr/bin/generate-token runs every time. The fix is to add creates: pointing at a sentinel file the task creates, so Ansible skips the task if the file exists.
• Templated files written without a notification check. A task that copies a templated config and notifies a handler will run the handler on every apply if the template content happens to be identical but the timestamp differs. The fix is to use the template module’s built-in change detection, which compares content rather than timestamps.
• shell re-running database migrations or other side-effecting commands. The fix is to record the migration in a state file and use creates: or when: to skip it on subsequent runs.

How to detect idempotency in CI

molecule test runs the idempotency check as part of its meta-step sequence. After the initial converge, Molecule re-runs converge against the same ephemeral target and asserts that the second run reports zero changes. A role that fails this step is a role that cannot be safely re-applied.

Locally, the same property can be observed with check mode:

ansible-playbook --check --diff playbook.yml

The --check flag tells Ansible to report what it would do without doing it. The --diff flag tells Ansible to show the file-level changes that would be applied. A role that is idempotent will, on a host already converged, report changed=0 for every task. A role that is not idempotent will report a non-zero changed= count, and --diff will show which tasks would mutate the host.

The discipline is to run --check --diff against staging after every successful Molecule scenario and before the production apply. If the staging check reports changes, the role is not idempotent on staging and the production apply should be blocked.

Change detection at the fleet level

Idempotency answers “will this role re-apply cleanly?”. Change detection answers “is this fleet still in the state the role produced?”. The two questions are different, and a production discipline needs both.

Change detection is implemented as a scheduled run of the playbook in check mode against the production fleet:

ansible-playbook --check --diff -i inventories/production playbook.yml

A scheduled job (cron, GitHub Actions schedule, Jenkins) runs this command against the production inventory every hour or every day. The output is captured and compared against the previous run. A non-empty changed= count is a drift signal: something on the fleet has changed since the last apply, and the role would now mutate the host to bring it back to the desired state.

The four signals change detection surfaces:

• Manual drift. An operator SSHed to a host and edited a config file by hand. The next check-mode run reports the change.
• External drift. A package was updated by the OS package manager outside of Ansible. The next check-mode run reports the package version mismatch.
• Accidental drift. A partial apply left a host in an intermediate state. The next check-mode run reports the unfinished work.
• Emergency drift. A break-glass change was applied to remediate an incident and was never backfilled into the role. The next check-mode run reports the unmanaged configuration.

Each of these signals is something a production team needs to see. A pipeline that runs change detection against production on a schedule is a pipeline that catches all four classes of drift within the detection interval.

What idempotency plus change detection enables

Three operational properties become possible when both disciplines are in place:

• Safe re-apply after partial failure. A role that fails halfway through an apply can be re-applied without compounding the damage. The second apply picks up where the first left off and converges the host.
• Drift as a ticket, not an outage. A change-detection schedule that surfaces drift as a ticket allows the team to remediate on their own schedule rather than discovering the drift during an incident.
• Audit trail of who changed what. A change-detection log that records the host, the task, and the timestamp answers the question “when did this configuration file change?” without needing to correlate system logs.

The properties are cumulative. A team that has idempotency but no change detection knows the role is safe to re-run but does not know whether the fleet is still in the desired state. A team that has change detection but no idempotency sees the drift but cannot re-apply the role to remediate it without producing further changes. Both are required.

Production discipline

1. Idempotency is enforced in Molecule. A role that fails the second-converge check does not merge.
2. Idempotency is verified on staging. A role that passes Molecule but fails the staging check-mode run does not promote to production.
3. Change detection runs on a schedule against production. The interval is a team decision; the schedule itself is non-negotiable.
4. Drift signals become tickets. A change-detection run that reports changed> 0 produces an alert; it does not silently accumulate.
5. The apply job’s idempotency is observable. Every apply reports its changed= count and the count is published to the audit log.

Cross-course references

• Ansible for Production Sysadmins - Part XXV (CheckDiff) covers the --check and --diff semantics in detail, including which modules respect check mode and which do not.
• Ansible for Production Sysadmins - Part XXIV (Idempotency) covers the module-by-module rules for writing idempotent tasks.
• This course, Part LXI (FailureHandling) - lesson git-cicd-gitops-lxi-05 covers the relationship between idempotency and safe retry.
• This course, Part LXXV (Drift) - lessons git-cicd-gitops-lxxv-01 through git-cicd-gitops-lxxv-06 cover drift detection and remediation for GitOps-managed systems; this lesson is the Ansible-specific instantiation of the same discipline.

Quiz