Lifecycle stages and hosts you must not touch · Ansible

Environment, site and role are the dimensions people build first, because they are stable. Lifecycle is the one people leave out, because it is not: a host moves through it several times in its life, and the movements are operational events rather than design decisions.

Leaving it out does not remove it. It relocates it — into a colleague’s memory, into a comment on a ticket, into a when: clause somebody added during an incident and nobody removed.

The lifecycle dimension

A host is always in exactly one of these states, and the state determines what automation is allowed to do to it:

Stage	Meaning	What automation may do
`building`	Provisioned, not yet configured	Everything. There is nothing to break.
`burn_in`	Configured, under test, no real traffic	Everything, including disruptive tests.
`live`	Serving production traffic	Only changes that respect the rollout rules.
`draining`	Being removed from service, still up	Read-only, or the drain procedure itself.
`decommissioned`	Retired, may still answer SSH	Nothing.

The value of writing it down is that it converts a judgement into a lookup. “Can I reboot web-a2?” stops being a question you ask in a chat channel and becomes a group membership that a play can read.

It also gives every play a sane default target. live is not the same set as all, and most plays want live:

Read-only / Safewhat a fleet-wide play actually should reach— Everything currently in service, with the two exception groups removed. Resolves the pattern and exits without connecting.

$ ansible -i inventory/production 'all:!maintenance_hold:!do_not_patch' --list-hosts

  hosts (2):
  web-a1.example.com
  web-a2.example.com

Exceptions are inevitable

Every estate over about fifty hosts has machines that are deliberately different:

The database that must not be rebooted this quarter because the vendor is investigating a firmware fault.
The appliance whose package set the vendor owns, where a routine apt upgrade voids support.
The host running the one application that has not been migrated, which needs an ancient library the standard build removed.
The machine somebody is actively debugging, right now, and must not have its configuration converged out from under them.

None of these are failures of discipline. They are operations. The failure of discipline is in how they get recorded.

How an exception becomes a snowflake

The undocumented version has a predictable shape:

# Somebody adds this during an incident. It is correct at the time.
- name: Apply security updates
  ansible.builtin.package:
    name: '*'
    state: latest
  when: inventory_hostname != 'db-a3.example.com'

Six months later, db-a3 has missed twenty patch cycles. Nobody knows why the exclusion is there, so nobody dares remove it. The person who added it has changed teams. It is now a snowflake, and the only record of its existence is one line in a file nobody greps.

Worse, the exclusion is invisible to every tool you have. It is not in the inventory, so --list-hosts still shows db-a3 in the target set. The run still reports ok for it. The blast-radius answer is wrong and the recap agrees with the wrong answer.

Exception groups instead

Make the exception a group. It costs the same three lines and it is visible to everything.

# inventory/production/hosts.yml
all:
  children:
    live:
      hosts:
        web-a1.example.com:
    draining:
      hosts:
        web-a2.example.com:
    maintenance_hold:
      hosts:
        db-a1.example.com:
    do_not_patch:
      hosts:
        appliance-a1.example.com:

# inventory/production/group_vars/maintenance_hold.yml
---
exception_reason: 'Vendor investigating a storage firmware fault; no reboots.'
exception_ticket: 'OPS-4471'
exception_owner_team: data-platform
exception_expires: '2026-06-30'

Four fields, each answering a question that will be asked:

exception_reason — why this host is different, in a sentence somebody unfamiliar can act on.
exception_ticket — where the decision was made and the discussion lives.
exception_owner_team — who to ask whether it still applies.
exception_expires — the date on which the exception stops being assumed and starts being questioned.

The expiry is the field that does the work. Without it, an exception is permanent by default, and permanence by default is exactly how a temporary vendor hold becomes a machine three years behind on patches.

Making the expiry mean something

An expiry date that nothing reads is a comment. Because group_vars resolve at run time, a play can check them, and because assert runs on the controller it can be a CI job that never touches a managed node.

---
- name: Exceptions must be documented and must expire
  hosts: maintenance_hold:do_not_patch
  gather_facts: false
  tasks:
    - name: Every exception carries a reason, a ticket, an owner and an expiry
      ansible.builtin.assert:
        that:
          - exception_reason is defined
          - exception_ticket is defined
          - exception_owner_team is defined
          - exception_expires is defined
        fail_msg: '{{ inventory_hostname }} is in an exception group with incomplete metadata'
        quiet: true

    - name: Expired exceptions must be renewed or removed
      ansible.builtin.assert:
        that:
          - exception_expires >= now(utc=true, fmt='%Y-%m-%d')
        fail_msg: >-
          Exception for {{ inventory_hostname }} expired on
          {{ exception_expires }} (ticket {{ exception_ticket }},
          owner {{ exception_owner_team }}). Renew it or remove the host
          from the exception group.
        quiet: true

Run it after the expiry date has passed:

Read-only / Safean exception that outlived its justification— Both assertions are controller-side. No connection is made to any managed node.

$ ansible-playbook -i inventory/production exception-audit.yml

TASK [Every exception carries a reason, a ticket, an owner and an expiry] ******
ok: [db-a1.example.com]
ok: [appliance-a1.example.com]

TASK [Expired exceptions must be renewed or removed] ***************************
fatal: [db-a1.example.com]: FAILED! => {"assertion": "exception_expires >= now(utc=true, fmt='%Y-%m-%d')", "changed": false, "evaluated_to": false, "msg": "Exception for db-a1.example.com expired on 2026-06-30 (ticket OPS-4471, owner data-platform). Renew it or remove the host from the exception group."}
ok: [appliance-a1.example.com]

PLAY RECAP *********************************************************************
appliance-a1.example.com   : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
db-a1.example.com          : ok=1    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

The now() call is evaluated on the controller at run time, and the %Y-%m-%d format makes a plain string comparison correct for ISO dates, so no date parsing is needed.

Notice what this changes politically as much as technically. Renewing the exception is now a commit: somebody has to change a date, and somebody else has to approve it. That is a much healthier default than an exclusion that renews itself by being ignored.

Under the hood

Exception groups work because Ansible resolves a host’s variables from every group it belongs to, not just the one the play targeted. A host in production, web, live and maintenance_hold carries the variables of all four into any play, whatever pattern selected it.

That is what makes a second, defensive style possible. Rather than relying on every play to remember the exclusion pattern, a play can refuse at the task level:

- name: Apply security updates
  ansible.builtin.package:
    name: '*'
    state: latest
  when: exception_reason is not defined

Two things to weigh before adopting it. It is more robust, because it travels with the task rather than with the invocation. But it is also silent: the host is skipped, the recap counts it under skipped, and somebody scanning for failures sees none.

The course’s position is to use the pattern exclusion as the primary control — because --list-hosts then tells the truth about blast radius — and to add the task-level guard only where the consequence of a missed exclusion is severe. Where you do add it, make the skip loud: pair it with a debug naming the host and the ticket, so the exception appears in the run log rather than only in the skipped count.

Decommissioned hosts: keep or delete?

There is a genuine trade-off here and the course will not pretend otherwise.

Keeping them in a decommissioned group means the hostname still resolves to something, so a stale reference in an old runbook produces a host in an inert group rather than a confusing “no hosts matched”. It also preserves the record that the machine existed, which matters during audit.

Deleting them means the inventory describes only what exists, so the host count is the truth and nobody can accidentally target a machine that was returned to the leasing company.

The workable compromise is time-boxed: move retired hosts into decommissioned with a removal date recorded the same way an exception expiry is, and delete them from the inventory when that date passes. The group is a staging area, not a graveyard. What you must not do is leave them in live because deleting entries feels risky — a decommissioned host inside live is a host that fleet-wide plays will keep trying to reach, and it will train your team to ignore unreachable hosts in the recap.

What happens if somebody adds a host to an exception group without the group_vars file?

The host is excluded from every play that excludes the group, and the audit above fails on the first assertion — exception_reason is defined is false — so CI rejects the change before it merges.

That is the intended behaviour, and it is why the metadata assertion is worth having even though it looks like paperwork. Without it, an exception group is just another way to hide a host, with the advantage over a when: clause that at least --list-hosts shows the truth, and the same disadvantage that nobody knows why.

The failure to design for is different and quieter: somebody adds the host to the group, writes the group_vars file, and sets exception_expires five years out. The audit passes. Guard against it in review rather than in code — an expiry longer than one change cycle is a decision that needs a person to agree with it, and a linter that caps the horizon will simply teach people to set it to the cap.