Running the window · Ansible

A maintenance window is a control if it can stop the change. If nothing that happens inside it can result in the change not proceeding, it is not a control — it is a calendar entry with an on-call rota attached.

That distinction is the whole lesson. Most windows fail it, not because anybody decided they should, but because no phase of the window was given the authority to say no.

Five phases

Phase	What it does	What it produces
Pre-check	Asserts the conditions the change assumes	A pass, or a cancelled window
Change	Applies the change, in batches, with gates	A changed fleet, or an abort
Validation	Proves the outcome, not the tasks	Evidence, or a rollback trigger
Rollback trigger	Fires the reverse when criteria are met	A restored fleet
Reporting	Records what happened, host by host	The input to the next window

Each phase can stop the window. That is the property that makes them phases rather than a checklist.

Phase 1 — pre-check

The pre-check play asserts everything the change assumes and refuses to let the window open if any of it is false. It changes nothing, so it can be run repeatedly — an hour before, and again at the top of the window.

Read-only / Safepreflight.yml— Executed on ansible-core 2.21.3 against a ten-host inventory. Nothing here writes to a target; ping and assert are both read-only.

- name: Pre-flight gate for the change window
hosts: appservers
gather_facts: true
any_errors_fatal: true
vars:
  change_window_id: CHG-2026-08-14-001
  max_backup_age_hours: 24
tasks:
  - name: Every targeted host answers
    ansible.builtin.ping:

  - name: Change freeze is not in force
    ansible.builtin.assert:
      that:
        - not freeze_active | bool
      fail_msg: "Change freeze active - {{ change_window_id }} may not proceed"
      quiet: true
    run_once: true

  - name: Root filesystem has room for the change
    ansible.builtin.assert:
      that:
        - (ansible_facts['mounts'] | selectattr('mount', 'eq', '/')
           | map(attribute='size_available') | first) > 2147483648
      fail_msg: "Less than 2 GiB free on / on {{ inventory_hostname }}"
      quiet: true

  - name: Backup on this host is fresh enough to be a rollback
    ansible.builtin.assert:
      that:
        - backup_age_hours | int < max_backup_age_hours | int
      fail_msg: >-
        Backup on {{ inventory_hostname }} is {{ backup_age_hours }}h old,
        limit is {{ max_backup_age_hours }}h
      success_msg: "Backup {{ backup_age_hours }}h old"

Run against ten hosts where one has a stale backup, this is what a blocked window looks like:

Read-only / Safethe pre-check refusing the window— Executed on ansible-core 2.21.3. web03 has a backup 216 hours old against a 24-hour limit. Output trimmed to the relevant hosts.

$ ansible-playbook -i inventories/prod preflight.yml; echo "exit=$?"

TASK [Every targeted host answers] *********************************************
ok: [web01]
ok: [web02]
ok: [web03]

TASK [Change freeze is not in force] *******************************************
ok: [web01] => {
  "changed": false,
  "msg": "All assertions passed"
}

TASK [Backup on this host is fresh enough to be a rollback] ********************
ok: [web01] => {
  "changed": false,
  "msg": "Backup 3h old"
}
fatal: [web03]: FAILED! => {
  "changed": false,
  "msg": "Backup on web03 is 216h old, limit is 24h"
}

PLAY RECAP *********************************************************************
web01                      : ok=3    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
web02                      : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
web03                      : ok=1    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

exit=2

Exit 2. The window does not open. One host out of ten has a backup that predates the last week, so the reverse procedure for that host is not available, so the change does not run — on any host, because any_errors_fatal: true makes one host’s failure the play’s failure.

Phase 2 — the dry run against the exact host list

The dry run has to target the host list the window will use. Not the group, not the pattern from last time, not “prod minus the ones we know about”. The exact list.

Read-only / Safeestablish the host list, then dry-run it— --list-hosts resolves the pattern without connecting. Every subsequent command in the window uses the same --limit string, unchanged.

# 1. What does the pattern actually resolve to?
ansible-playbook -i inventories/prod site.yml \
--limit 'appservers:!maintenance' --list-hosts

# 2. What tasks will run?
ansible-playbook -i inventories/prod site.yml \
--limit 'appservers:!maintenance' --list-tasks

# 3. What would change? Read the diff, not the recap.
ansible-playbook -i inventories/prod site.yml \
--limit 'appservers:!maintenance' --check --diff

The reason for the discipline: a --limit that differs by one host between the dry run and the real run means the dry run examined a different change. That sounds pedantic until the extra host is the one with the hand-edited config.

--diff in a window prints file contents, including secrets

--check --diff is the most useful command in a pre-change sequence and the easiest way to publish every managed secret on the estate.

The diff shows the content being written. If any managed file contains a credential, a key or a token, that content goes to the terminal, and from there to the window log, the screen share, and the ticket someone pasted it into.

Mitigations, in order of preference: no_log: true on the tasks that handle secret material, keeping secrets out of templated files entirely in favour of a runtime lookup, and — as a last resort — running the diff only against a subset that has no secret files. Part XXI covers the mechanics; the point here is that the window is exactly when this happens, because a window is when people run --diff and share their screen.

Phase 3 — the go/no-go gate

An explicit decision, made by a named person, after reading the pre-check result and the dry run, and before the first task that changes anything.

It is worth being blunt about what this is defending against. The default behaviour of a maintenance window is that the change proceeds because the window has started. The gate exists to make proceeding an act rather than an absence.

Three things make a gate real:

A named decider, who is present. Not a role, a person.
Written inputs: the pre-check exit code, the dry-run change count, the confirmation that the reverse procedure and its backups exist.
The ability to say no with no cost. If cancelling at the gate requires an apology, the gate will always say yes.

The gate is also where the timebox is agreed out loud: this window ends at 03:30, and at 03:30 an incomplete change is rolled back rather than continued. Fixing the end time before starting is what stops the “we are nearly there” spiral at 04:15.

Phase 4 — validation, and the trigger

Validation is a separate play that runs after each batch, and it tests outcomes rather than tasks. That distinction has its own lesson in Part XLIX; the window-level rule is simpler:

A validation failure is a change failure. Not a note for the morning.

The validation play exits non-zero, the abort criteria have been met, and the reverse procedure runs. Making that automatic rather than discretionary is lesson 5.

Phase 5 — reporting

The report is written because the next window is planned from it.

Field	Why it is in the report
Host list, as resolved	The next window starts from a known population
Per-host outcome	Distinguishes changed, unchanged, failed, never attempted
Batch timings	Turns “about an hour” into a number for the next plan
Deviations from the plan	The most valuable field and the most often omitted
Whether rollback was used, and how long it took	Validates or refutes the estimate in the plan
Hosts left in an intermediate state	The input to lesson 7

Why the pre-check uses any_errors_fatal and the change play uses max_fail_percentage

They are different questions, and using the wrong one produces a gate that does not gate.

any_errors_fatal: true means one host failing ends the play for every host. In the run above, web03 failed its backup assertion and the play stopped — even though nine hosts passed. That is correct for a gate, because the question a gate answers is “is the estate ready”, and the answer is no if any part of it is not.

The change play asks a different question: “is this change going badly enough to stop”. One host out of two hundred failing to converge is usually noise; forty is an incident. That is a proportion, which is what max_fail_percentage expresses.

There is a timing detail in any_errors_fatal that surprises people. It does not abort the instant a host fails — Ansible completes the current task across all hosts in the play or batch, then ends. In the pre-check output above, every host ran the backup assertion and reported its result before the play stopped. That is usually what you want in a gate: you learn about all ten failures, not just the first one, so the window can be rescheduled once rather than three times.

The two keywords also interact, and the interaction is worth knowing before writing both into the same play. Verified by execution on ansible-core 2.21.3: a play with serial: 5, any_errors_fatal: true and max_fail_percentage: 90 — a threshold that would tolerate one failure in five — stopped anyway on the single induced failure, and the second batch never started. any_errors_fatal wins, so max_fail_percentage in the same play is decoration.

What happens if the pre-check fails on one host out of two hundred?

This is the question that decides whether the gate survives contact with an actual estate, and the answer is not “always cancel”.

Three responses, all legitimate, and the plan should say in advance which one applies:

Cancel the window. Correct when the failure suggests the estate is not in the state the change assumes — a freeze flag set, a prior run left half-applied, the artefact missing from the repository. These are fleet-level facts and one host discovering them is enough.
Exclude the host and proceed. Correct when the failure is local and the host can safely be left out: it is unreachable, or its backup is stale. Add it to the exclusion, re-run the pre-check so the recorded host list is accurate, and record the host as deferred. The critical part is the re-run — an excluded host that is still in the plan’s host list produces a report that does not match reality.
Fix the condition and re-run the pre-check. Correct when the failure is a stale backup that can be retaken in ten minutes, and the window has ten minutes.

What must not happen is the fourth response: noting the failure, deciding it is probably fine, and proceeding with the original host list. That converts the gate into a formality, and everyone present learns that it is one.

Knowledge check

Why this matters

Maintenance windows drift into ritual quietly. The pre-check keeps passing, so people stop reading it. The dry run keeps showing the same changes, so people stop diffing it. The gate keeps saying go, so it stops being asked. None of these is a decision anybody made; each is what happens when a control produces the same output twenty times in a row.

The test for whether a window is still a control is a single question, and it is worth asking out loud at the gate: what would we have seen tonight that would have stopped this?

If nobody can answer, the window is not protecting the change. It is scheduling it. The phases in this lesson exist so that there is always an answer — a failing assertion, a diff nobody expected, a batch that validated badly — and so that the answer arrives before the fleet does.