Guardrails operators will not bypass

Every guardrail in this part can be removed from a run by one command-line option, and the operator who removes it will have a perfectly good reason.

That is not a flaw to be engineered away — a system in which no human can ever override the automation is a system that will eventually be overridden by a human with vim and no audit trail. The design problem is narrower and more interesting: make the guard so rarely wrong that nobody develops the habit of skipping it, and make the skip, when it happens, deliberate and recorded.

How a guard disappears

Start with a guard that carries a tag, which is how most of them are written:

- name: Refuse to run outside production
  ansible.builtin.assert:
    that:
      - deploy_env == 'production'
    fail_msg: "Refusing: deploy_env is '{{ deploy_env }}'."
  tags: [preflight]

- name: The change itself
  ansible.builtin.debug:
    msg: 'would deploy'
  tags: [deploy]

It works when the playbook is run normally. There are two ways it stops working, and only one of them looks deliberate.

$ ansible-playbook -i inv.ini deploy.yml --skip-tags preflight; echo "exit=$?"

TASK [The change itself] *******************************************************
ok: [localhost] => {
  "msg": "would deploy"
}

PLAY RECAP *********************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

exit=0

That one at least reads as a decision: somebody typed --skip-tags preflight. The second way is worse.

$ ansible-playbook -i inv.ini deploy.yml --tags deploy; echo "exit=$?"

TASK [The change itself] *******************************************************
ok: [localhost] => {
  "msg": "would deploy"
}

PLAY RECAP *********************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

exit=0

Identical output. Nobody skipped a guard — they ran the deploy tag, which is a normal, sensible, documented thing to do. The guard was not skipped so much as never invited, and there is nothing in the run to indicate it exists.

skipped=0 is the detail worth pausing on. A tag-deselected task does not appear in the skipped count at all, so there is no number anywhere in that recap that changes when a guard vanishes.

always, and its one hole

Tagging a guard always selects it regardless of --tags:

$ ansible-playbook -i inv.ini deploy.yml --tags deploy; echo "exit=$?"

TASK [Refuse to run outside production] ****************************************
fatal: [localhost]: FAILED! => {"assertion": "deploy_env == 'production'", "changed": false, "evaluated_to": false, "msg": "Refusing: deploy_env is 'staging'."}

PLAY RECAP *********************************************************************
localhost                  : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

exit=2

That closes the silent hole. It does not close the deliberate one:

$ ansible-playbook -i inv.ini deploy.yml --skip-tags always; echo "exit=$?"

TASK [The change itself] *******************************************************
ok: [localhost] => {
  "msg": "would deploy"
}

PLAY RECAP *********************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

exit=0

always is a normal tag with special selection behaviour, not a protected keyword. --skip-tags always removes it.

Which is fine — that argument is unmistakable in a shell history and impossible to type by accident. It is the difference between an override and an oversight, and that is the only distinction this design can achieve.

The placement that has no tag at all

For anything genuinely destructive, do not rely on tags:

# roles/database_restore/tasks/main.yml
- name: Verify preconditions before anything destructive
  ansible.builtin.import_tasks: preflight.yml

- name: Restore
  ansible.builtin.import_tasks: restore.yml

Note that this is only untaggable if the importing task carries no tag. Tags on import_tasks are inherited by every task in the imported file, so a tags: [preflight] on that first line hands the whole guard file to --skip-tags in one line.

A separate first play is often proposed as the stronger version. It is worth being exact about what it does and does not buy, because the intuition here is wrong:

# site.yml
- name: Preconditions
  hosts: all
  gather_facts: false
  any_errors_fatal: true
  tasks:
    - name: Confirm target, inputs and change window
      ansible.builtin.assert:
        that:
          - deploy_env | default('UNDECLARED') == 'production'
          - change_ticket | default('') | length > 0
        fail_msg: '...'
      tags: [always]        # <- this line is load-bearing

- name: The change
  hosts: webservers
  roles:
    - webserver

Without that tags: [always], the guard play is no protection against a tag selection at all. Tag arguments apply across the whole invocation, not per play, so an untagged guard in play one is deselected by --tags deploy exactly as an untagged guard in play two would be:

$ ansible-playbook -i inv.ini site.yml --tags deploy; echo "exit=$?"

PLAY [Preconditions] ***********************************************************

PLAY [The change] **************************************************************

TASK [The change itself] *******************************************************
ok: [web01.example.com] => {
  "msg": "changing web01.example.com"
}
ok: [web02.example.com] => {
  "msg": "changing web02.example.com"
}
ok: [web03.example.com] => {
  "msg": "changing web03.example.com"
}

exit=0

A play header with nothing under it is the only tell, and it is not a tell anybody reads.

So the separate play earns its place for two other reasons — ordering (every host is checked before any host is changed) and any_errors_fatal: true (one refusal stops the whole run) — and the protection against tag selection comes entirely from always. Use both; do not credit the play structure with the tag behaviour.

The real defence: a guard that is right

Every mechanism above is a lock, and locks are only as good as the tenant’s willingness to carry the key. The durable defence is a guard that does not fire on legitimate work.

A guard is characterised by two numbers:

The top-left cell is why you wrote it. The bottom-left cell is what determines whether it still exists in a year.

A guard that refuses one legitimate change in twenty teaches the whole team a habit: when this fires, add the flag. After that, the guard is decoration — it still appears in code review, it still looks like a control, and it is bypassed reflexively by everyone including the person who wrote it.

An override that leaves a trace

The override you design is competing with --skip-tags, so it has to be at least as convenient and it has to produce something --skip-tags does not: a record.

- name: Refuse to run outside production without a stated reason
  ansible.builtin.assert:
    that:
      - deploy_env | default('UNDECLARED') == 'production'
        or override_reason | default('') | length > 0
    fail_msg: >-
      Refusing: deploy_env is
      '{{ deploy_env | default('UNDECLARED') }}'. If this is
      deliberate, re-run with
      -e override_reason='CHG-12345 rebuild of the staging tier'
      and the override will be recorded.
    quiet: true
  tags: [always]

- name: Record that a guardrail was overridden
  ansible.builtin.debug:
    msg: >-
      GUARDRAIL OVERRIDE on {{ inventory_hostname }}: environment guard
      bypassed, reason: {{ override_reason }}
  when: override_reason | default('') | length > 0
  tags: [always]

$ ansible-playbook -i inv.ini deploy.yml -e '{"override_reason": "CHG-12345 rebuild of the staging tier"}'

TASK [Refuse to run outside production without a stated reason] ****************
ok: [localhost]

TASK [Record that a guardrail was overridden] **********************************
ok: [localhost] => {
  "msg": "GUARDRAIL OVERRIDE on localhost: environment guard bypassed, reason: CHG-12345 rebuild of the staging tier"
}

The override is now three things --skip-tags is not: it requires a reason, the reason appears in the run output where the log collector sees it, and it names the specific guard rather than removing every task that shares a tag.

Knowledge check