When new hosts appear on their own · Ansible

This is the lesson the part exists for.

Every safety practice in this course so far rests on an assumption so basic that it is rarely stated: that a change in what your automation does is preceded by a change you can see. Somebody edits a playbook. Somebody edits an inventory file. There is a commit, a diff, a reviewer, a timestamp.

A dynamic inventory breaks that assumption completely.

A new instance is created, tagged role=web, and joins the web group. Your playbook is unchanged. Your inventory config is unchanged. Your repository has no new commits. The next scheduled run configures a machine you have never seen, and there was nothing to review.

Nothing failed. Nothing was misconfigured. The system did precisely what it was built to do, and the blast radius grew.

The incident, in the order it actually happens

The version of this that reaches a post-mortem usually runs like this.

Monday. A patching playbook targets linux_servers and reboots what it patches. It has run every week for eight months without incident.

Tuesday. A different team stands up a new database server. They tag it according to the organisation’s tagging standard, which includes os=linux. They have never heard of your playbook. Nothing they did was wrong.

Wednesday, 02:00. The dynamic inventory maps os=linux into linux_servers. The new database joins. It is patched and rebooted during its first business day of production traffic.

Wednesday, 09:00. Somebody opens git log on the Ansible repository looking for what changed. Nothing changed. That is the part that costs the most time, because every instinct says a change must have caused this, and the evidence trail is not in git at all. It is in the inventory.

Defence one: keep the host list, and diff it

The host list is data. Snapshot it, and a change in the fleet becomes a diff like any other.

Read-only / Safesnapshot the effective host list— Read-only: --list-hosts resolves the pattern and prints the matches without connecting to anything. --flush-cache re-queries the provider, so you snapshot the fleet rather than the cache. tail -n +2 drops the hosts (N): header; the count is worth keeping separately.

SNAP_DIR=/var/lib/ansible/inventory-snapshots
mkdir -p "$SNAP_DIR"

ansible -i inventory/ 'linux_servers' --list-hosts --flush-cache | tail -n +2 | tr -d ' ' | sort > "$SNAP_DIR/today.txt"

Compare it with yesterday’s, and the fleet change is visible as text:

Read-only / Safeone host appeared— Real output from ansible-core 2.21.3 plus diff. The inventory config did not change between these two snapshots; the source did.

$ diff inventory-snapshots/yesterday.txt inventory-snapshots/today.txt

4a5
> web04.example.com

comm is better for scripting, because it separates additions from removals and gives you each as a plain list:

Read-only / Safeadditions only— Real output from GNU coreutils. -13 suppresses lines unique to file one and lines common to both, leaving only what is new today. Swap to -23 for what disappeared.

$ comm -13 inventory-snapshots/yesterday.txt inventory-snapshots/today.txt

web04.example.com

Make the delta fail something

A snapshot nobody reads is a file. The value comes from the comparison being enforced:

Read-only / Safean inventory delta gate— Read-only: it queries the inventory and compares two text files. It changes nothing and connects to nothing, but its exit code can stop a pipeline.

#!/usr/bin/env bash
set -euo pipefail

SNAP_DIR=/var/lib/ansible/inventory-snapshots
PATTERN=${1:?usage: inventory-gate.sh <pattern>}
PREV="$SNAP_DIR/previous.txt"
CURR="$SNAP_DIR/current.txt"

ansible -i inventory/ "$PATTERN" --list-hosts --flush-cache | tail -n +2 | tr -d ' ' | sort > "$CURR"

if [ ! -f "$PREV" ]; then
echo "no baseline; establishing one"
cp "$CURR" "$PREV"
exit 0
fi

added=$(comm -13 "$PREV" "$CURR")
removed=$(comm -23 "$PREV" "$CURR")

if [ -n "$added" ] || [ -n "$removed" ]; then
echo "inventory changed since the last approved snapshot"
[ -n "$added" ]   && printf 'ADDED:\n%s\n' "$added"
[ -n "$removed" ] && printf 'REMOVED:\n%s\n' "$removed"
exit 1
fi

echo "inventory unchanged: $(wc -l < "$CURR") hosts"

Three runs against a source that gained a host between the second and the third:

Read-only / Safebaseline, quiet, then the signal— Real output from ansible-core 2.21.3 driving the script above. The third run exits 1, which is what a scheduler or a pipeline stage will act on. Nothing in the Ansible repository changed between run two and run three.

$ inventory-gate.sh all; inventory-gate.sh all; inventory-gate.sh all

no baseline; establishing one
rc=0

inventory unchanged: 6 hosts
rc=0

inventory changed since the last approved snapshot
ADDED:
web06.example.com
rc=1

Two design decisions in that script are worth defending.

It fails on removals too, not only additions. A host disappearing is either a decommission you know about or a source returning partial results, and the second one is the more dangerous. Treating both directions as “tell me” costs nothing.

Updating the baseline is a deliberate act. previous.txt is only overwritten by a human — or by a pipeline step that requires an approval. That is what makes it a review gate rather than a log line. A script that silently rebaselines on every run has all the cost of this control and none of the benefit.

Defence two: opt in, do not opt out

The Monday-to-Wednesday incident above has a design cause, and it is the grouping rule.

Opt-out grouping: a host is in scope unless something excludes it. os=linux maps to linux_servers. Every new Linux machine anybody creates, anywhere in the account, is automatically in scope for everything that targets linux_servers. The default is “managed”.

Opt-in grouping: a host is in scope only if something explicitly includes it. A host joins linux_servers only when it carries ansible_managed=true, applied by someone who knows what that means. The default is “not managed”.

Read-only / Safethe same estate, two grouping rules

# Opt-out: everything Linux is in scope. New machines join silently.
groups:
linux_servers: provider_tags.os == 'linux'

# Opt-in: only machines explicitly enrolled are in scope.
groups:
linux_servers: >-
  provider_tags.os == 'linux'
  and provider_tags.get('ansible_managed', 'false') == 'true'

The cost of opt-in is real and worth stating honestly: a machine that should have been patched and was not tagged goes unpatched, and that is also an incident. You are choosing which failure you would rather have.

The reason production estates should still choose opt-in is asymmetry. A machine missing from automation is a gap you find with a reconciliation report comparing the provider’s full list against the managed list — the subject of the source-of-truth lesson in Part V, and a report you can run daily. A machine unexpectedly in automation is a reboot during business hours, and you find it by having caused it.

Defence three: treat the host list as a reviewable artefact

The deepest fix is to stop letting an unreviewed host list reach a change at all.

Query the inventory as a separate, earlier step than the change, and write the result to a file: ansible-inventory --list --flush-cache --limit <target-group>.
Diff that file against the last approved snapshot, and stop if it differs.
Attach the file to the change ticket. It is the answer to "which hosts" in a form that can be read, counted and disagreed with.
Convert it into a limit file and run the change with --limit @that-file, so the run targets the reviewed list rather than re-querying and possibly getting a different answer.
Update the approved snapshot as an explicit step after the change, not automatically.

Step four is the one that closes the loop, and it is the subject of Part XXX lesson 4. A run that re-queries the provider at execution time can target a host list that differs from the one reviewed ten minutes earlier. A run pinned to --limit @approved-hosts.txt cannot: the list was frozen at review time, and a host created since then is simply not in the file.

That is as close as a dynamic inventory gets to the property a static inventory has for free — that the thing you approved is the thing that runs.

Under the hood

Why can Ansible not just warn you that the host list grew?

Because it has no idea what you expected. Inventory is assembled from sources, patterns are resolved against it, and the result is a set of hosts. Nothing in that pipeline carries a memory of the previous run, an expected count, or a notion of “normal”. The plugin returned four hundred hosts today and three hundred yesterday, and both were correct answers to the question asked.

This is the same structural reason a pattern matching zero hosts is a warning rather than an error, and the same reason a pattern matching ten times too many hosts is not even a warning: Ansible knows what matched, and cannot know what you meant.

Which puts the expectation outside the tool by necessity. It lives in a snapshot file, in an assertion inside the play, in a number on a change ticket. All three are things you have to create; none of them can be inferred. Part XXX lesson 8 builds the in-play version, and it works by the same principle as the snapshot here — you state the number, and the automation refuses to exceed it.

What happens if the new host is one you genuinely should manage?

Then the delta gate fires, you look at the addition, you agree with it, and you update the baseline. Thirty seconds.

That is the honest cost of this control and it is worth being explicit about it, because the objection to inventory delta gates is always the same: in an estate that scales, the gate fires constantly and people learn to rebaseline without looking. A control that is always firing is a control nobody reads — the same failure as an alert that pages nightly.

Three adjustments keep it meaningful:

Scope the gate to the groups that matter. Autoscaling in role_web_stateless is routine. A new member of role_db_primary is not. Gate the second, log the first.

Gate on what the change targets, not on the whole inventory. If the playbook runs against linux_servers:&env_prod, snapshot exactly that pattern. Additions elsewhere are somebody else’s business.

Separate “new host” from “new host in a group with a reboot in its playbook”. Criticality is an axis, and it is the axis Part XXX lesson 1 opens with. A delta gate that treats all additions identically is measuring host count and ignoring what the hosts are.

Knowledge check

Why this matters

Reviewing code is a solved problem. Every organisation has a mechanism for it, and everybody understands that a change to a playbook is a change that gets looked at.

A dynamic inventory quietly moves part of the decision — which machines — outside that mechanism, into a system where the people making changes have never heard of your automation and are not doing anything wrong. The tagging standard they follow was written by someone else, for someone else’s purpose, and it is now an input to what your playbooks do.

Which means the host list needs the same treatment as code: it is snapshotted, it is diffed, a difference is a thing somebody looks at, and the version that was reviewed is the version that runs. None of that is expensive. All of it has to be built, because none of it is what you get by default.

The next two lessons finish the picture: how a static “never touch these” list stays authoritative over anything the API says, and what happens when the API answers wrongly rather than not at all.