Lab: The dry run that lied

Objective

By the end of this lab you will have a play whose --check --diff output is clean, reassuring and completely wrong, and you will have the real run’s failure beside it. You will then audit every task for check-mode support before running it, fix the gap, and write down the three things check mode still cannot tell you.

Architecture

A single play that deploys a small application. Everything runs against ansible_connection: local writing into a scratch directory, so no managed node is required to reproduce the core result.

scratch/
├── migrate.sql       <- rendered by a copy task
└── app.conf          <- rendered by a template task

/usr/bin/false        <- stands in for the migration tool that fails

Requirements

A controller with ansible-core 2.21.x. The exact output below was captured from 2.21.3.
No managed nodes needed. Every task runs against ansible_connection: local and writes only into scratch/. A real node is worth using if you want to extend the module audit to apt, dnf and a real systemd_service — hence B-nested as the higher-fidelity mode — but nothing in the lab as written requires one.
A writable scratch directory. Nothing outside it is touched.

Scenario

The change process at your organisation requires a --check run attached to every change ticket. Last week’s ticket had one: changed=3, failed=0, a clean diff of the config file, no warnings.

The real run failed on the third task, on all forty hosts, after the config file had been written. The change reviewer’s reasonable question — “what was the point of the dry run?” — is what this lab answers.

Tasks

Task 1: Build the play

WORKDIR="$HOME/ansible-checkmode-lab"
mkdir -p "$WORKDIR/scratch"
cd "$WORKDIR"

inventory.yml:

all:
  hosts:
    node1:
  vars:
    ansible_connection: local

deploy.yml:

- name: Deploy the reporting application
  hosts: all
  gather_facts: false
  vars:
    scratch: "{{ playbook_dir }}/scratch"

  tasks:
    - name: Render the migration script
      ansible.builtin.copy:
        content: |
          -- schema migration 004
          ALTER TABLE reports ADD COLUMN generated_at timestamptz;
        dest: "{{ scratch }}/migrate.sql"
        mode: '0644'

    - name: Run the migration
      ansible.builtin.command:
        cmd: "/usr/bin/false --apply {{ scratch }}/migrate.sql"

    - name: Record that the migration completed
      ansible.builtin.debug:
        msg: "migration complete"

/usr/bin/false stands in for a migration tool that exits non-zero. In the real ticket it was a psql invocation against a database that rejected the statement.

Task 2: Run the dry run and believe it

Read-only / Safecontroller— Check mode. Nothing is written; this is genuinely safe.

$ ansible-playbook -i inventory.yml deploy.yml --check --diff

PLAY [Deploy the reporting application] ****************************************

TASK [Render the migration script] *********************************************
--- before
+++ after: /home/operator/ansible-checkmode-lab/scratch/migrate.sql
@@ -0,0 +1,2 @@
+-- schema migration 004
+ALTER TABLE reports ADD COLUMN generated_at timestamptz;

changed: [node1]

TASK [Run the migration] *******************************************************
skipping: [node1]

TASK [Record that the migration completed] *************************************
ok: [node1] => {
  "msg": "migration complete"
}

PLAY RECAP *********************************************************************
node1                      : ok=2    changed=1    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0

failed=0. A clean diff showing exactly the file that will be written. A final task confirming the migration completed.

Every word of that is true and the conclusion a reviewer draws from it is false.

skipping: [node1] is the whole story, and it scrolls past

TASK [Run the migration] reports skipping. Not “would run”, not “cannot verify” — skipping, the same word Ansible uses for a task excluded by a when: condition.

The command module cannot predict what an arbitrary command would do, so under check mode it does not run it and reports skipped. The skipped=1 in the recap is the only trace, and skipped=1 is such an ordinary thing to see that nobody reads it as “the most dangerous task in this play was not evaluated”.

Confirm the file really was not written, so you know check mode itself is behaving correctly:

ls -la scratch/

Empty. Check mode did exactly what it promised. It simply promised less than the reviewer assumed.

Task 3: Run it for real

Configuration changecontroller— Writes into the scratch directory and then fails

$ ansible-playbook -i inventory.yml deploy.yml

TASK [Render the migration script] *********************************************
changed: [node1]

TASK [Run the migration] *******************************************************
fatal: [node1]: FAILED! => {"changed": true, "cmd": ["/usr/bin/false", "--apply", "/home/operator/ansible-checkmode-lab/scratch/migrate.sql"], "delta": "0:00:00.002", "msg": "non-zero return code", "rc": 1, "stderr": "", "stdout": ""}

PLAY RECAP *********************************************************************
node1                      : ok=1    changed=1    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

Illustrative output

The file exists. The migration did not run. The third task — the one that said “migration complete” in the dry run — never executed.

ls -la scratch/

Put the two recaps side by side in comparison.md. ok=2 changed=1 failed=0 skipped=1 against ok=1 changed=1 failed=1 skipped=0. Nothing in the first predicts the second.

Task 4: Audit the play before you run it

The fix begins with a habit: for every task in a play you are about to check-mode, ask what its module’s check_mode attribute says. ansible-doc answers this directly.

for m in copy command shell package systemd_service uri template lineinfile debug; do
  printf '%-18s ' "$m"
  ansible-doc "$m" 2>/dev/null \
    | sed -n '/ATTRIBUTES:/,/EXAMPLES:/p' \
    | grep -A4 'check_mode:' \
    | grep -E 'support:' \
    | head -1
done

Read-only / Safecontroller— Check-mode support per module, read from the modules themselves

$ ansible-doc command | sed -n '/ATTRIBUTES:/,/EXAMPLES:/p' | grep -A4 'check_mode:'

        check_mode:
      description: Can run in check_mode and return changed status prediction without modifying
        target, if not supported the action will be skipped.
      details: while the command itself is arbitrary and cannot be subject to the check
        mode semantics it adds `creates'/`removes' options as a workaround
      support: partial

Note the sentence in the description: “if not supported the action will be skipped”. That is the documented behaviour, stated plainly, and it is what produced the clean dry run.

The results, verified on 2.21.3:

Module	`check_mode` support	Consequence under `--check`
`copy`, `template`, `lineinfile`, `blockinfile`	full	predicts the change, shows a diff
`systemd_service`, `apt`	full	predicts service and package state
`command`, `shell`	partial	skipped, unless `creates:`/`removes:` is set
`package`	full, with a caveat	“support depends on the underlying plugin invoked”
`service`	full, with a caveat	“support depends on the underlying plugin invoked”
`uri`	none	skipped
`debug`, `assert`	n/a	run normally

Write the audit into check-support.md for every module your play uses.

Task 5: Make the play honest

Three techniques, applied to the three kinds of task.

A read-only command: run it in check mode. check_mode: false forces a task to execute even under --check. It is safe only if the task cannot change anything:

    - name: Confirm the migration tool is present and reports a version
      ansible.builtin.command: /usr/bin/false --version
      register: tool
      changed_when: false
      failed_when: false
      check_mode: false      # safe: --version reads and exits

Now the dry run tells you something real: whether the tool exists.

A command with a predictable side effect: give it creates:. With creates:, command becomes a stat comparison and gains real check-mode support:

    - name: Run the migration
      ansible.builtin.command:
        cmd: "/usr/bin/false --apply {{ scratch }}/migrate.sql"
        creates: "{{ scratch }}/.migration-004.done"

Under --check the module stats the marker file and reports changed or ok accordingly instead of skipping. It still cannot tell you the command will succeed — nothing can — but the dry run now says “this would run” rather than “skipped”.

The task that cannot be predicted: say so out loud. Where a task genuinely cannot be evaluated, make the play announce it rather than let skipped=1 carry the message:

    - name: State plainly what this dry run did not evaluate
      ansible.builtin.debug:
        msg: >-
          CHECK MODE: the migration itself was not executed and its success
          is not predicted by this run. Tasks not evaluated:
          'Run the migration', 'Verify the API responds'.
      when: ansible_check_mode

Re-run the dry run and compare:

ansible-playbook -i inventory.yml deploy.yml --check --diff | tail -20

Task 6: The second class of gap — order dependence

Check mode does not only skip tasks. It also evaluates every task against the current state, because no earlier task actually changed anything.

Add a task that reads back what a previous task wrote:

    - name: Render the base configuration
      ansible.builtin.copy:
        content: "listen = 8080\n"
        dest: "{{ scratch }}/app.conf"
        mode: '0644'

    - name: Read back the rendered configuration
      ansible.builtin.slurp:
        src: "{{ scratch }}/app.conf"
      register: rendered

    - name: Assert the port directive is present
      ansible.builtin.assert:
        that: "'listen' in (rendered.content | b64decode)"
        success_msg: "config verified"

Run it in check mode against a scratch directory where app.conf does not exist yet:

Read-only / Safecontroller— The verification task fails in check mode and succeeds for real

$ ansible-playbook -i inventory.yml verify-order.yml --check

TASK [Render the base configuration] *******************************************
changed: [node1]

TASK [Read back the rendered configuration] ************************************
fatal: [node1]: FAILED! => {"changed": false, "msg": "File not found: /home/operator/ansible-checkmode-lab/scratch/app.conf"}

Then run it for real — ok=3 changed=1 failed=0 — and run it in check mode a second time, now that the file exists: ok=3 changed=0 failed=0.

A check-mode failure is not necessarily a real failure

The copy task reported changed — it would write the file — but it did not write it, so slurp found nothing and the play failed.

For real, all three tasks succeed. The check run is a false positive, and it disappears the moment the host is converged.

This is the mirror image of the first problem, and the two together give the rule: check mode is a drift detector, not a deployment rehearsal. On an already-converged host it answers “does the current state differ from the declared state” accurately. On a fresh host it is unreliable in both directions — silently optimistic about the tasks it skips, and spuriously pessimistic about the ones that depend on tasks it did not perform.

The same failure comes from lineinfile with create: false on a file an earlier task would have created, from a command reading a rendered file, and from any assert on a registered read. It is not an edge case; it is what a build-from-nothing play looks like under --check.

In blind-spots.md, record the three classes of thing check mode structurally cannot catch. From this lab:

Tasks it skips. command and shell without creates:/removes:, uri entirely, and anything whose module reports support: none. The only trace is skipped=N.
Order-dependent effects. A task whose input is a previous task’s output, on a host that is not already converged — a slurp of a file copy would have written, a lineinfile with create: false. Fails in check mode, succeeds for real.
Anything that depends on the command actually running. A migration that would deadlock, a package that would conflict, a service that would fail its post-start check. Check mode predicts state changes; it does not execute anything, so it cannot observe a runtime failure.

Validation

ansible-playbook -i inventory.yml deploy.yml --check --diff reports failed=0 with skipped=1, and ls scratch/ shows nothing was written.
The same play run for real reports failed=1 after writing scratch/migrate.sql.
ansible-doc command | grep -A4 check_mode reports support: partial and names creates/removes as the workaround.
ansible-doc uri | grep -A4 check_mode reports support: none.
With creates: added, the migration task under --check reports changed or ok rather than skipping.
With the ansible_check_mode task added, the dry-run output ends with an explicit statement of what was not evaluated.
verify-order.yml --check fails at the slurp task on a fresh scratch directory, succeeds for real (ok=3 changed=1), and then passes in check mode on the second attempt (ok=3 changed=0).
check-support.md covers every module the play uses.

Expected Outcome

ansible-checkmode-lab/
├── blind-spots.md
├── check-support.md
├── comparison.md
├── deploy.yml
├── inventory.yml
└── scratch/

You can predict, before running a dry run, which of its tasks will be skipped. Your plays end with a statement of what check mode did not evaluate. And you can say why a clean --check on a change ticket is evidence of something narrower than the reviewer assumes.

Troubleshooting

--check reports changed on every task, every time. Check mode against a host that is not converged reports every difference between current and declared state — which is correct and is what makes it a drift detector. Run it against a converged host to get a meaningful answer.

check_mode: false on a task that writes. This is the dangerous inversion: the task now executes during a --check run, so your “dry run” changes the host. Only ever set it on a task you can state, in one sentence, cannot change anything.

check_mode: true — the reverse — is also available. It forces a task into check mode even in a real run, which is how you make a specific destructive task never execute while the rest of the play does. Useful and easy to forget you set.

ansible_check_mode is undefined. It is defined on every run, true or false. If a template reports it undefined, the template is being rendered by something outside a task context — a lookup('template', ...) on the controller, for example.

A package task behaves differently to the audit. package delegates to the platform’s manager, and its documentation says so explicitly: support depends on the underlying plugin invoked. Audit the concrete module — apt, dnf, zypper — not the abstract one.

--diff shows nothing for a template task in check mode. The task is being skipped for a different reason — usually a when: — or the file is already identical. -v distinguishes the two.

Cleanup

Everything this lab wrote is inside scratch/ in the working directory. No managed node was contacted for Tasks 1–5, and nothing outside the working directory was modified.

Step 1. Confirm that, rather than assuming it:

cd "$HOME/ansible-checkmode-lab"

# Nothing should exist outside scratch/ that the lab created
find . -newer inventory.yml -type f | sort

# And nothing at all outside the working directory
ls -la /usr/bin/false      # untouched, it is a system binary the lab only ran

Step 2. If you extended the module audit against a real managed node and installed anything to do it, remove it only if it was not already present.

Step 3. Keep the audit and remove the directory:

rm -rf deletes the whole lab directory, including anything you added

The next block removes ~/ansible-checkmode-lab and everything under it, recursively and without confirmation. If you saved notes, an inventory you want to keep, or a playbook you extended, copy it out first — the step above does that for the lab’s own deliverables and knows nothing about yours.

The path is written in full and anchored to $HOME deliberately. A relative rm -rf run from the wrong directory is how people delete work they meant to keep, and the shell gives no second chance.

Confirm what you are about to delete before you delete it:

ls -la "$HOME/ansible-checkmode-lab"

mkdir -p "$HOME/ansible-lab-deliverables"
cp -a "$HOME/ansible-checkmode-lab/check-support.md" \
      "$HOME/ansible-checkmode-lab/blind-spots.md" \
      "$HOME/ansible-lab-deliverables/"

rm -rf "$HOME/ansible-checkmode-lab"

Under the hood

Check mode is a flag passed to every module in its arguments, _ansible_check_mode: true. What happens next is entirely up to the module.

A module whose check_mode attribute is support: full reads it, performs its comparison against the current state, and returns the changed it would have produced without calling the write path. copy checksums, file stats, apt queries the package database. These are cheap because the comparison is the same one the module does anyway before deciding to act.

A module with support: none — uri is the canonical example — declares that it cannot make that prediction. The action plugin sees this in the module’s metadata and skips the task without invoking it, which is why the output says skipping rather than something more specific. There is no mechanism for a module to say “I cannot predict this but here is a warning”; the vocabulary is binary.

command sits between the two with support: partial. Its documentation explains the mechanism precisely: “Check mode is supported when passing creates or removes. If running in check mode and either of these are specified, the module will check for the existence of the file and report the correct changed status. If these are not supplied, the task will be skipped.” The workaround is not a special case — it converts the unpredictable execution into a predictable stat, which is something the module can evaluate without running anything.

check_mode: false on a task strips the flag before the module sees it, so the module has no way to know a --check run is in progress. That is why it is a genuine hazard on a task that writes: nothing downstream will stop it.

What You Learned

A clean --check can precede a failing real run, and you have both recaps side by side to prove it.
skipping is the only signal, and it is indistinguishable from a when: exclusion. skipped=1 in a recap deserves the same attention as failed=1.
ansible-doc <module> | grep -A4 check_mode is the audit. You produced a per-module support table from the modules themselves rather than from folklore.
uri has no check-mode support, so every health-check-based safety argument evaporates in a dry run.
creates: converts an unpredictable command into a predictable stat, which is why it is command’s documented check-mode workaround.
Check mode is a drift detector, not a rehearsal. On a converged host it is accurate; on a fresh one it produces false failures from order dependence.
when: ansible_check_mode lets a play state its own blind spots, which is the smallest change that makes a dry run honest evidence.

Objectives

Prerequisites