Guardrails: refusing to run too wide · Ansible

Everything in this part has been a habit: print the list, read the count, compare it against a number. Habits work until the night they do not — the incident at 03:00, the new joiner, the one run that skipped the wrapper because somebody was in a hurry.

This lesson turns the habit into machinery. A pre-flight play that travels with the playbook and refuses, before anything changes, when the target set is too large, contains a protected host, or reaches production without somebody having said so out loud.

Part XXIV built the case for guardrails and the “wrong fleet” guard. This is the targeting-specific one, assembled from the variables in the previous lesson, and every refusal below is real output.

The pre-flight play

Read-only / Safepreflight.yml - four refusals— Read-only throughout: assert connects to nothing and changes nothing. It is a separate play so it runs once for the playbook rather than once per serial batch.

- name: Pre-flight - refuse to run too wide
hosts: all
gather_facts: false
vars:
  max_hosts: 5
  protected_group: do_not_automate
  confirm_production: false
tasks:
  - name: The target set must not be empty
    run_once: true
    ansible.builtin.assert:
      that: ansible_play_hosts_all | length > 0
      fail_msg: 'Refusing: the target set is empty.'

  - name: The target set must not exceed the declared ceiling
    run_once: true
    ansible.builtin.assert:
      that: ansible_play_hosts_all | length <= max_hosts | int
      fail_msg: >-
        Refusing: this run targets {{ ansible_play_hosts_all | length }} hosts,
        ceiling is {{ max_hosts }}. Narrow it with --limit, or raise max_hosts
        deliberately.
      success_msg: '{{ ansible_play_hosts_all | length }} hosts, within the ceiling of {{ max_hosts }}'

  - name: The target set must not intersect the protected group
    run_once: true
    vars:
      protected_hit: "{{ ansible_play_hosts_all | intersect(groups[protected_group] | default([])) }}"
    ansible.builtin.assert:
      that: protected_hit | length == 0
      fail_msg: 'Refusing: {{ protected_hit | join(", ") }} are in {{ protected_group }}.'

  - name: Production requires an explicit acknowledgement
    run_once: true
    vars:
      prod_hit: "{{ ansible_play_hosts_all | intersect(groups['production'] | default([])) }}"
    ansible.builtin.assert:
      that: prod_hit | length == 0 or confirm_production | bool
      fail_msg: >-
        Refusing: {{ prod_hit | length }} production hosts targeted without
        -e confirm_production=true.

Four properties of that play are deliberate and worth naming.

hosts: all, not the group the change targets. The guard must see whatever the operator asked for, so it takes the widest possible pattern and lets --limit narrow it exactly as it narrows the real play. A guard scoped to web cannot notice that somebody limited to db.

A separate play, not a first task. run_once under serial means once per batch, so a guard placed inside a batched play runs several times — harmless for an assertion, harmful for anything with a side effect. As its own play with no serial, it runs exactly once.

groups[...] | default([]) so a missing group is an empty list rather than an undefined-variable error. A guard that crashes when the protected group has not been created yet is a guard people delete.

confirm_production defaults to false. The safe value is the default, and the unsafe one has to be typed. Never the other way round.

It refuses, verified

Read-only / Safeno limit at all - ten hosts against a ceiling of five— Real output from ansible-core 2.21.3, exit code 2. The empty check passed, the ceiling check failed, and the run stopped before the play that changes anything.

$ ansible-playbook -i inventory/hosts.yml preflight.yml; echo exit=$?

TASK [The target set must not be empty] ****************************************
ok: [web01.example.com] => {
  "msg": "All assertions passed"
}

TASK [The target set must not exceed the declared ceiling] *********************
fatal: [web01.example.com]: FAILED! => {
  "assertion": "ansible_play_hosts_all | length <= max_hosts | int",
  "evaluated_to": false,
  "msg": "Refusing: this run targets 10 hosts, ceiling is 5. Narrow it with --limit, or raise max_hosts deliberately."
}
exit=2

Read-only / Safenarrowed to the canary - but it is a production host— Real output from ansible-core 2.21.3, exit code 2. The count is fine and the host is not protected; what stops it is that production was reached without an acknowledgement.

$ ansible-playbook -i inventory/hosts.yml preflight.yml --limit canary; echo exit=$?

    "msg": "1 hosts, within the ceiling of 5"
...
fatal: [web01.example.com]: FAILED! => {
  "msg": "Refusing: 1 production hosts targeted without -e confirm_production=true."
}
exit=2

Read-only / Safethe acknowledgement supplied - all four checks pass— Real output from ansible-core 2.21.3, exit code 0. Four assertions, all green, and the real play would now run.

$ ansible-playbook -i inventory/hosts.yml preflight.yml --limit canary -e confirm_production=true

PLAY RECAP *********************************************************************
web01.example.com          : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Read-only / Safea protected host, with the acknowledgement supplied— Real output from ansible-core 2.21.3, exit code 2. The acknowledgement does not unlock the protected group - that is a separate check, and it names the host it refused on.

$ ansible-playbook -i inventory/hosts.yml preflight.yml --limit do_not_automate -e confirm_production=true

    "msg": "1 hosts, within the ceiling of 5"
...
fatal: [db01.example.com]: FAILED! => {
  "msg": "Refusing: db01.example.com are in do_not_automate."
}

Make it a role, then it is one line per playbook

Wrap the four tasks in a role — roles/preflight/tasks/main.yml — and every playbook gains the guard with:

- name: Pre-flight
  hosts: all
  gather_facts: false
  roles:
    - preflight

max_hosts then comes from defaults/main.yml at a conservative value and is overridden per playbook where a larger radius is legitimate. A patching playbook that genuinely targets four hundred hosts sets max_hosts: 450 in its own vars, which is a deliberate, reviewable, greppable statement rather than an absence of a guard.

What the guard cannot do

Being precise about this matters, because a guard believed to cover more than it does is worse than none.

It cannot catch zero hosts. With an empty target there is nothing for the task to run on, so the “must not be empty” assertion never executes — the play reports skipping: no hosts matched and the run proceeds to the next play. That check is there for the case where a later play empties out, and the genuinely-zero case needs the wrapper from lesson 6.

It cannot stop a run that skips it. ansible-playbook site.yml without the pre-flight play in front of it is unguarded. That is why the role form matters: the guard belongs inside the playbook it protects, not beside it. import_playbook at the top of site.yml is the strongest version, because then there is no invocation that omits it.

It cannot know your intent. max_hosts is a number somebody chose. Set it too high and it never fires; set it too low and people learn to pass -e max_hosts=9999, which is a routed-around guard and the subject of the second guardrails lesson in Part XXIV.

`order:` — and the value that quietly breaks canarying

The order: play keyword controls the sequence in which hosts are processed, which under serial decides which hosts are in the first batch. Five values, all verified:

Read-only / Safeall five order values, serial: 1— Real output from ansible-core 2.21.3, one run per value, showing execution sequence. inventory and sorted agree here only because the inventory happens to be in sorted order.

$ ansible-playbook -i inventory/hosts.yml order.yml

order: inventory          web01  web02  web03  web04
order: sorted             web01  web02  web03  web04
order: reverse_sorted     web04  web03  web02  web01
order: reverse_inventory  web04  web03  web02  web01
order: shuffle            web04  web02  web03  web01

An invalid value is a hard error rather than a fallback, which is the right behaviour:

Read-only / Safea typo'd order value— Real output from ansible-core 2.21.3. The run stops rather than silently defaulting to inventory order.

$ ansible-playbook -i inventory/hosts.yml order.yml

[ERROR]: Invalid 'order' specified for inventory hosts: bogus

`shuffle` and canarying are mutually exclusive

A canary is an argument of the form: this change worked on host X, so it will probably work on the rest. The argument depends on knowing which host X was, and on X being the same host each time so that “succeeded on the canary” means something across runs.

order: shuffle destroys both halves. The first batch is different on every run, so a successful canary yesterday tells you nothing about today’s first host; and if a rollout fails after two batches, the set of already-changed hosts is different every time, which makes recovery a per-incident investigation rather than a procedure.

shuffle has one legitimate use — spreading load on a shared backend so the same hosts do not always go first — and it is only safe when every host is genuinely interchangeable and no batch is being used as a canary. In a rollout that stages by risk, it is actively harmful.

If you want a specific canary, do not rely on ordering at all. Put the host in a canary group and run that group as its own invocation.

The full pre-execution sequence

Confirm the inventory source and its freshness: ansible-config dump --only-changed, and --flush-cache if the source is cached and the run changes state.
Print the effective host list with the exact command you will run: ansible-playbook <pb> --limit <pattern> --list-hosts. Read the count out loud.
Confirm the count against the change ticket. A mismatch stops here, before anything else.
Confirm the task set: --list-tasks, with any --tags you intend to pass.
Run --check --diff and read the diff, remembering it is evidence rather than proof.
Confirm the stop procedure: how you will notice, how you will halt, and what recovers the hosts already changed.
Run, with the pre-flight guard in front and the first batch small enough that being wrong is survivable.

This is the checklist the rest of the course refers back to. It is seven lines and most of it is read-only.

Under the hood

Why an assert rather than fail with a when:?

Both stop the run. assert is preferable for a guard for three reasons, and they are all about what happens after it fires.

The failure output names the expression. The recap includes "assertion": "ansible_play_hosts_all | length <= max_hosts | int" and "evaluated_to": false, so whoever hit the guard sees the rule as well as the message. A fail reports only the message you wrote.

that: is a list, and every element must hold. Several related conditions can live in one task without a compound Jinja expression that nobody can read.

It carries success_msg. A guard that says 1 hosts, within the ceiling of 5 on a normal run is a guard people trust, because they have watched it work. A guard that is silent until the day it fires is a guard people assume is broken and bypass.

The exit code is worth knowing precisely: a failed assertion is a task failure, so ansible-playbook exits 2, not 1. A wrapper checking for exit 1 specifically will treat a fired guardrail as success. Check for non-zero, or check for 2 explicitly if you want to distinguish a guard refusal from an argument error.

One structural note: the guard reads groups, which is the whole inventory, not just the play’s hosts. That is what lets a play targeting web ask a question about do_not_automate without targeting it — groups is populated from inventory regardless of the pattern.

What happens if somebody raises max_hosts to get past the guard?

Then the guard did its job, and what happens next is a process question rather than a technical one.

Passing -e max_hosts=9999 is not itself illegitimate. There are real runs that need a large radius, and a guard with no override is a guard that blocks legitimate work — which is how guards get deleted rather than overridden. The distinction that matters is whether the override leaves a trace.

An override in the playbook’s own vars is a commit. It is reviewed, it is greppable, and git log says who raised it and when.

An override on the command line is invisible after the terminal closes. It exists only in shell history on one machine.

So the shape to aim for is: the guard has an override, the override is a variable, and the variable’s normal home is a file. A CI job that greps for -e max_hosts in its own command line and refuses is a cheap way to push the override into the reviewable place without forbidding it.

The deeper version of this problem — guards that fire on legitimate work and therefore get routed around — is the subject of the second Part XXIV guardrails lesson. The short form: a guard that fires often is a guard that is wrong, and the fix is usually the threshold rather than the override.

Knowledge check

Why this matters

This part opened with a definition — blast radius is the number and criticality of the systems a mistake can affect — and everything since has been a way of knowing that number before it matters.

The pattern syntax decides what a target expression means. --limit narrows it and cannot widen it. --list-hosts proves the result. Zero hosts is a warning in one case and an error in another, and the difference is worth knowing at three in the morning. And the scope variables let a play state its own limits, so that the last line of defence is inside the thing being run rather than in a wrapper somebody skipped.

The guard in this lesson is not clever. It is four assertions and a number somebody chose. What makes it valuable is that it is present on every invocation, that it says what it refused and why, and that it turns “I thought this only hit the canary” from an incident report into an error message printed before anything happened.

Everything after this part — serial batches, rolling deployments, patching, disaster recovery — assumes you can answer “who does this hit” from the command line. You can now.