Recording an approved deviation

Some hosts have to be different, and the difference is correct.

db07 runs the billing batch job alongside the web tier, so eight nginx workers starve it. A regulated environment requires a log retention that differs from the standard. A device from a vendor will not tolerate the sysctl everything else gets.

Every estate has these. Almost none of them have a mechanism for recording them, which is why the same three findings appear in every drift report and get manually dismissed every time, until the report stops being read.

The three things people do instead

All three are worse than the mechanism, in different ways.

A conditional in the role.

nginx_worker_processes: "{{ 2 if inventory_hostname == 'db07' else 8 }}"

The value is right. Everything else about it is wrong: the reason is absent, the owner is absent, there is no expiry, it will still be there in four years, and finding all such conditionals requires reading every role. It is also the exact pattern the next lesson treats as the definition of a snowflake.

Removing the host from the group. Nobody has to think about db07 again, because db07 is now unmanaged in its entirety. One setting needed to differ; instead the host lost coverage of the other 179 declarations, and it will not appear in any drift report, any compliance statement, or any patching run.

This is the most damaging option and it is the one that gets chosen at 02:00, because it makes the noise stop immediately.

Nothing. The finding recurs nightly. Somebody dismisses it. Six months later everybody dismisses everything.

An exception is data

The mechanism is to make the exception a piece of inventory data that carries both the value and the justification, and to have the role consume it.

config_exceptions:
- id: EX-2026-014
  expires: '2026-11-30'
  owner: platform-team
  ticket: OPS-4471
  reason: >-
    Shares hardware with the billing batch job. Eight nginx workers
    starve the batch during month-end close. Reduced until the batch
    moves to its own host under OPS-4102.
  vars:
    nginx_worker_processes: 2

The role does not know anything about db07. It reads nginx_worker_processes as it always did. What changes is that the declared state for this host is now 2, so a converged db07 reports changed=0 — correctly, because 2 is what it is supposed to be.

That is the conceptual move worth pausing on: an exception is not a suppression of a finding. It is a change to the declaration for one host, with the reasoning attached.

- name: apply approved, unexpired exceptions
ansible.builtin.set_fact:
  "{{ item.key }}": "{{ item.value }}"
loop: >-
  {{ config_exceptions | default([])
     | selectattr('expires', 'gt', now(fmt='%Y-%m-%d'))
     | map(attribute='vars') | list | combine | dict2items }}
loop_control:
  label: "{{ item.key }}"

$ ansible-playbook -i inventory.ini exceptions.yml

TASK [apply approved, unexpired exceptions] ************************************
ok: [localhost] => (item=nginx_worker_processes)

TASK [show effective values] ***************************************************
ok: [localhost] => {
  "msg": "workers=2 keepalive=65"
}

nginx_keepalive was the subject of the expired exception and stayed at the standard 65. Nobody deleted anything; the expiry did it.

Expiry is the load-bearing field

Without it, exceptions are permanent, and a permanent exception is indistinguishable from an undeclared requirement that nobody ever promoted.

Two behaviours make expiry real.

An expired exception stops applying, shown above — the host reverts to the standard, which is the outcome you want if nobody could be bothered to renew it. It also means expiry has a blast radius: on the day an exception lapses, the next enforcement run changes that host.

An expired exception is reported. Silent lapsing is only half the job; somebody should be told before it bites.

vars:
renewal_horizon: "{{ '%Y-%m-%d' | strftime((now().timestamp() | int) + 2592000) }}"

- name: warn about exceptions expiring within 30 days
ansible.builtin.debug:
  msg: >-
    {{ item.id }} ({{ item.owner }}) expires {{ item.expires }}
    - {{ item.ticket }}
loop: "{{ config_exceptions | default([]) }}"
loop_control:
  label: "{{ item.id }}"
when:
  - item.expires > now(fmt='%Y-%m-%d')
  - item.expires < renewal_horizon

- name: fail the audit if any exception has lapsed
ansible.builtin.assert:
  that:
    - config_exceptions | default([])
      | rejectattr('expires', 'gt', now(fmt='%Y-%m-%d'))
      | list | length == 0
  fail_msg: >-
    Lapsed exceptions on {{ inventory_hostname }}. Renew them with a new
    approval, or delete them and let the host converge to the standard.

$ ansible-playbook -i inventory.ini expiry-check.yml

TASK [list expired exceptions] *************************************************
skipping: [localhost] => (item={'id': 'EX-2026-014', 'expires': '2026-11-30'})
ok: [localhost] => (item={'id': 'EX-2025-003', 'expires': '2025-06-30'}) => {
  "msg": "EXPIRED: EX-2025-003 (expired 2025-06-30)"
}

TASK [fail if any exception has expired] ***************************************
fatal: [localhost]: FAILED! => {
  "msg": "expired exceptions present"
}

Reviewing the whole set

Individual exceptions are fine. The population is the health metric, and it is trivially reportable because they are all data in one shape:

cat > exception-register.py <<'PY'
import json, sys
hv = json.load(sys.stdin)['_meta']['hostvars']
rows = [(h, e) for h, v in hv.items() for e in v.get('config_exceptions', [])]
rows.sort(key=lambda r: r[1]['expires'])
for host, e in rows:
  print(f"{e['expires']}  {e['id']:<14} {e['owner']:<16} {host:<28} {e['ticket']}")
print(f"{len(rows)} exceptions across {len({h for h, _ in rows})} hosts")
PY

ansible-inventory -i inventory/ --list | python3 exception-register.py

What to look at in that report:

• Total count, trended. Rising steadily means the standard no longer fits the estate, and the standard is what should change.
• Exceptions sharing a reason. Four hosts excepted for the same cause is not four exceptions; it is a missing group, and it should become one with its own group_vars.
• Ownerless or ticketless entries. These will not survive review, and they are usually the oldest ones.
• Clustering on one host. A host with eleven exceptions is not an excepted host; it is a different kind of host, and the next lesson argues about what to do with it.

Knowledge check