Staged validation and idempotency — check mode, diff mode, and the production pattern

The final gate before the apply on production is the staged validation pattern: run the playbook against the target inventory in check mode with diff enabled, observe what would change, and only proceed to the apply if the diff is what the change was expected to produce. The pattern works because of a property that distinguishes Ansible from imperative configuration tools: idempotency. A playbook that is idempotent can be run repeatedly; the first run makes the changes the playbook describes, and every subsequent run is a no-op. Idempotency is what makes the apply step a routine execution rather than a leap of faith.

What —check and —diff do

The two flags operate at the task level and answer two different questions:

• --check runs the playbook without executing the action. For most modules, the module is loaded, its argument spec is validated, and its check_mode: true code path returns what the change would be without making it. The task is reported as changed or ok based on the dry-run result.
• --diff produces a unified diff of the changes that would be made. For file-modifying modules, the diff shows before-and-after for the affected file. For non-file modules, the diff is empty or shows parameter changes.

The two flags are independent. --check without --diff says “this is what would change” without showing the actual content. --diff without --check runs the playbook for real and shows what changed. The production combination is both:

ansible-playbook --check --diff playbook.yml

This produces the full plan: which tasks would report changed, and for each one, the actual content change that would be applied.

flowchart LR
    A["ansible-playbook --check --diff"] --> B[For each task]
    B --> C[Load module in check mode]
    C --> D[Compute would-change]
    D --> E[For file modules, compute diff]
    E --> F{More tasks?}
    F -->|yes| B
    F -->|no| G[Print summary]

The output is the review artefact. A reviewer who reads the combined --check --diff output sees exactly what the apply will do to the target inventory without the apply having happened.

What —check deliberately does not do

Check mode is not a simulation; it is a dry run. The distinction matters:

• No state mutation. Tasks that would write files, install packages, restart services, or modify users do none of those things in check mode.
• No module side effects. Tasks that talk to external systems (cloud APIs, databases, message brokers) report what they would do but do not call out.
• Incomplete coverage for modules that do not honour check mode. Some modules do not implement check_mode: true and report “skipped” or assume “changed” without inspecting actual state. The playbook author must read each module’s documentation to know whether check mode is honoured.
• No idempotency proof from a single —check run. Check mode proves what would change on a single run. Idempotency is a property of a second run; proving it requires Molecule’s idempotence stage or a manual second apply.

The gate’s value is that it surfaces the plan before the apply, not that it executes the plan. A reviewer reading the output can decide whether the planned changes match the change’s intent before any system is mutated.

Idempotency: the property that makes repeated applies safe

Idempotency is the property that a playbook run on a target in the desired state produces no changes. A second run is a no-op. A tenth run is a no-op. The first run after drift produces the changes needed to return to the desired state, and the run after that is a no-op again.

This property is what allows:

• Re-runs after partial failure. A playbook that crashes halfway through can be re-run; the second run applies the missing changes without re-applying the successful ones.
• Convergence over time. A playbook can be run on a cron, applying drift remediation on every schedule.
• Safe retries. A flaky network or SSH connection can be retried; the playbook will apply only what was missed.
• Confident applies. The apply on production can proceed knowing that the same playbook will be a no-op on the next run, which means any mistake in the playbook will be visible in the first run rather than accumulating across runs.

A playbook that is not idempotent - that is, a playbook whose modules use command/shell without changed_when, or that uses creates/removes incorrectly, or that touches files with timestamps that always differ - cannot be safely re-run. The first run applies the change; the second run applies it again, possibly with side effects. Production Ansible requires idempotency.

The Molecule scenario’s idempotence stage is the gate that proves this property: it converges, then converges again, and asserts that the second run reported zero changes. A scenario without this stage has not tested idempotency.

The staged validation pattern

The full production pipeline, in order:

1. yamllint . - YAML shape, milliseconds, no host.
2. ansible-lint - Ansible shape, seconds, no host.
3. ansible-galaxy collection install -r collections/requirements.yml - install the collections the playbook references.
4. ansible-playbook --syntax-check playbook.yml - parse the playbook end-to-end.
5. ansible-playbook --check --diff playbook.yml - run against the target inventory in dry-run mode with diff. This is the plan that the human review reads.
6. molecule test - converge, assert idempotence, verify against ephemeral hosts. The strongest runtime claim.
7. The apply on production - ansible-playbook playbook.yml against the production inventory, with appropriate become, with the runner identity authorised for the change.

Steps 1-4 are static gates with no network contact. Step 5 is a read-only network operation that produces a plan. Step 6 is a runtime claim against disposable infrastructure. Step 7 is the actual change.

The discipline is that every gate before step 7 must pass, and the diff in step 5 must match the change’s intent. If any earlier gate fails, step 7 does not run. If the diff in step 5 is unexpected, the change is revised and the gates rerun. Step 7 is then a routine execution of an already-validated plan, not a leap of faith.

What the diff in —check mode looks like

For a file-modifying task like ansible.builtin.template, the combined --check --diff output shows the source template, the destination path, and the unified diff between the existing file and the would-be file:

TASK [Configure nginx listen port] *
--- before
+++ after
@@ -1,3 +1,3 @@
 # Managed by Ansible
-listen 80;
+listen 8080;

The reviewer reads this and decides whether the change matches intent. A diff that shows the wrong port is the signal to revise the template and re-run the gates; a diff that shows the right port is the signal to proceed.

For non-file tasks, the diff is empty but the task still reports changed or ok. The decision is in the task summary: a run that reports many changed tasks where the change was supposed to touch one file is a signal that the playbook has unintended side effects.

Idempotency pitfalls in practice

Common patterns that break idempotency:

• command/shell without changed_when. A shell command that runs every time and reports changed unconditionally is not idempotent. The fix is a changed_when expression that inspects the output.
• creates/removes on a command that touches files outside the named path. The creates argument prevents re-runs only if the named path exists; if the command writes files elsewhere, those files are written again on every run.
• Templates with non-deterministic content. A template that includes { ansible_date_time.iso8601 } is non-idempotent by construction.
• Modules that ignore check mode. A module that does not implement check_mode: true may report changed on every run. The fix is module replacement or an explicit changed_when.

The Molecule idempotence stage catches all of these by asserting that the second run produces no changes. A scenario whose second run reports changes is a scenario that has caught a real bug.

Production discipline

1. --check --diff runs against the real target inventory in CI. Running it only against a staging inventory hides drift between staging and production.
2. The diff output is part of the PR review. A reviewer who approves the code without reading the diff has not reviewed the change.
3. Molecule’s idempotence stage is not optional. A scenario that disables it to make CI faster has stopped testing the property that production requires.
4. Modules that ignore check mode are flagged in code review. The playbook author must justify the choice; the default expectation is that the module honours check mode.
5. The apply on production runs only after every prior gate has passed. Skipping a gate to “save CI time” is how unvalidated changes reach production.
6. A playbook that is not idempotent is blocked from the default branch. Production Ansible requires idempotency; a non-idempotent playbook is not production code.

Cross-course references

• Ansible for Production Sysadmins - Part XXV (CheckDiff) covers the mechanics of --check and --diff in depth; this lesson is the pipeline integration of those flags.
• Ansible for Production Sysadmins - Part XXVI (Testing) covers the Molecule idempotence stage that proves the property.
• Ansible for Production Sysadmins - Part XXXVI (Drift) covers the operational use of check mode for drift detection after the change is applied.
• Ansible for Production Sysadmins - Part XXXVIII (GitCI) is the full pipeline; this lesson is the final gate before the apply.

Quiz