Guarding against the wrong fleet

The single most expensive Ansible incident is not a broken playbook. It is a correct playbook, run against the wrong hosts, by someone who believed they were running it against the right ones.

It is worth being precise about how that happens, because “be careful with -i” only addresses the first of four routes.

Only the first is an operator error. The other three are correct commands, run correctly, that reach a fleet nobody chose — which is why the control has to live in the automation rather than in the operator.

The guard that does not work

This is the first thing everybody writes, and it is worse than nothing:

# Do not do this
- name: Confirm we are on production
  ansible.builtin.assert:
    that:
      - deploy_env | default('production') == 'production'
    fail_msg: 'Not production.'

The default('production') means the assertion passes on every host that never declared an environment at all. A brand-new host, a host whose group_vars file was not merged, a host in the wrong inventory that simply has no deploy_env — all of them sail through a guard whose entire purpose was to stop them.

The general shape of the mistake: a guard that is satisfied by the absence of evidence is not a guard. Where a default is unavoidable, make it the value that fails:

- deploy_env | default('UNDECLARED') == 'production'

Now a host that says nothing is refused, and the operator finds out that the host is undeclared rather than being told it is production.

Two independent sources must agree

A single variable is one file away from being wrong. The stronger guard requires two facts that come from different places to say the same thing:

- name: Confirm the target fleet from two independent sources
  ansible.builtin.assert:
    that:
      - deploy_env | default('UNDECLARED') == 'production'
      - "'production' in group_names"
    fail_msg: >-
      Refusing: deploy_env is
      '{{ deploy_env | default('UNDECLARED') }}' and this host belongs
      to groups {{ group_names }}. Both must say production before this
      play will change anything.
    success_msg: >-
      Target confirmed: {{ inventory_hostname }} is in {{ group_names }}
      with deploy_env={{ deploy_env }}.

group_names is the list of groups the host belongs to, computed from the inventory itself. deploy_env comes from group_vars/production.yml. They are different files maintained by different mechanisms, and a mistake in either one is caught by the other.

$ ansible-playbook -i inventories/production/hosts deploy.yml

TASK [Confirm the target fleet from two independent sources] *******************
ok: [web01.example.com] => {
  "changed": false,
  "msg": "Target confirmed: web01.example.com is in ['production', 'webservers'] with deploy_env=production."
}
ok: [web02.example.com] => {
  "changed": false,
  "msg": "Target confirmed: web02.example.com is in ['production', 'webservers'] with deploy_env=production."
}

The evidence for the guard is visible before the run, without executing anything:

$ ansible-inventory -i inventories/production/hosts --graph

@all:
|--@ungrouped:
|--@production:
|  |--@webservers:
|  |  |--web01.example.com
|  |  |--web02.example.com

A blast-radius ceiling

Environment guards answer “am I on the right kind of host”. They do not answer “am I on more hosts than this procedure was written for”, and that is a different mistake with a different cause — usually a pattern that matched more than intended, or an inventory that grew.

ansible_play_hosts_all holds every host the play resolved to, before any failures or batching:

- name: Refuse a run wider than this procedure is approved for
  ansible.builtin.assert:
    that:
      - ansible_play_hosts_all | length <= max_hosts | int
    fail_msg: >-
      Refusing: this play resolved to
      {{ ansible_play_hosts_all | length }} hosts, which is more than
      the {{ max_hosts }} this procedure is approved for. Narrow it with
      --limit, or raise max_hosts deliberately and record why.
    quiet: true
  run_once: true
  vars:
    max_hosts: 2

$ ansible-playbook -i inv.ini restricted.yml

TASK [Refuse a run wider than this procedure is approved for] ******************
fatal: [web01.example.com]: FAILED! => {"assertion": "ansible_play_hosts_all | length <= max_hosts | int", "changed": false, "evaluated_to": false, "msg": "Refusing: this play resolved to 3 hosts, which is more than the 2 this procedure is approved for. Narrow it with --limit, or raise max_hosts deliberately and record why."}

run_once: true is deliberate. The question is about the play, not about each host, so evaluating it two hundred times produces two hundred identical failures and one useful one.

Where the guard lives decides whether it survives

A guard in the playbook protects that playbook. A guard in the role protects everything that imports the role, including the playbook somebody writes next year for a purpose you did not anticipate.

roles/
  database_restore/
    tasks/
      main.yml        <- imports preflight.yml first, always
      preflight.yml   <- the guards
      restore.yml     <- the destructive work

# roles/database_restore/tasks/main.yml
- name: Verify preconditions before anything destructive
  ansible.builtin.import_tasks: preflight.yml

- name: Restore
  ansible.builtin.import_tasks: restore.yml

The property this buys is specific: there is no way to use the destructive part without the guards. Not “the documentation says to run the check first”, not “the playbook we normally use includes the check” — no path exists. A new playbook that does roles: [database_restore] inherits the protection whether or not its author knew about it.

That is the sense in which a guardrail belongs in the role rather than in the operator’s memory. Memory is not inherited.

Knowledge check