Where the truth about the fleet lives · Ansible

Underneath every question this part has asked — what groups exist, who owns them, what all reaches — sits a question nobody writes down:

When the inventory file and reality disagree, which one is wrong?

Most estates have never answered it explicitly, which means they have answered it implicitly and differently in different situations. This lesson is about making the choice on purpose. It is deliberately about ownership rather than implementation; inventory plugins, caching and staleness are the subject of a later part.

Three candidate authorities

The Git inventory is authoritative

The inventory files in the repository define the fleet. A machine that exists but is not in the inventory is, by policy, not a managed machine — and probably should not exist.

What you gain: every membership change is a commit, with an author, a reviewer and a date. The whole apparatus of code review applies to “which machines are in production”, which is a remarkably strong control and costs nothing extra once the repository exists.

What you pay: freshness. A host provisioned at 14:00 is not in scope until somebody opens a pull request, and under pressure that step gets skipped. Estates that provision frequently find this model fights them.

Where it fits: physical fleets, long-lived VMs, anything where machines are created deliberately and rarely.

The CMDB or asset system is authoritative

An external system of record holds the fleet, and Ansible reads from it.

What you gain: one answer shared with everyone else who needs it — finance, security, capacity planning, the audit. The inventory stops being an Ansible-specific artefact.

What you pay: the review boundary moves out of Git and into the CMDB’s change process, which is frequently weaker — often a web form with no approval step. You have not removed the need for review; you have relocated it somewhere you may not control. And many CMDBs are maintained by hand, which makes them a second thing that drifts rather than a source of truth.

Where it fits: organisations that already have a CMDB people genuinely maintain, and a regulatory reason to have one.

The infrastructure provider is authoritative

The cloud API, the hypervisor, or the virtualisation platform is queried directly, and group membership derives from tags or attributes.

What you gain: accuracy about existence. Nothing is ever missing, because the thing that creates machines is the thing being asked.

What you pay: two things, and the second is easy to miss. First, a host enters scope the moment it is created — before anyone has reviewed whether it should be automated. Second, group membership now depends on tags, so a tagging mistake is a targeting mistake. Someone who mistypes an environment tag has moved a machine between environments with no commit, no review and no diff.

Where it fits: elastic estates where the machine population genuinely changes hourly and a Git-based inventory would be permanently wrong.

The two failure states

Whatever you choose, exactly two things can go wrong, and they are opposites.

The shadow host

A machine running in production that no inventory knows about.

It is not patched, because the patching play never targets it. It is not monitored, if monitoring is deployed by the same automation. It is not backed up. It does not appear in the audit, so its unpatched OpenSSL is not in the vulnerability report either. It works perfectly, quietly, until it does not.

Shadow hosts come from ordinary events: a machine provisioned by hand during an incident, a proof-of-concept that became load-bearing, a host rebuilt under a new name where the old name was never removed and the new one never added.

The ghost entry

An inventory entry for a machine that was decommissioned last quarter.

Its direct harm is small — it is unreachable in every run. Its indirect harm is large, and it is the reason this failure gets its own name: a fleet-wide run that always reports two unreachable hosts trains everybody to skip the unreachable count. When a real host goes down, the recap says three instead of two, and nobody notices the difference.

Reconciliation is the deliverable

Choosing an authority does not keep the inventory correct. A scheduled comparison does, and it must report both directions, because the two failure states are found by opposite comparisons.

Get the inventory’s view as a sorted host list:

Read-only / Safewhat automation believes exists— Expands the whole inventory to a plain sorted host list suitable for comparison.

ansible -i inventory/production all --list-hosts \
| tail -n +2 | tr -d ' ' | sort > /tmp/from-inventory.txt

Get the authority’s view the same way — a cloud CLI, a hypervisor API query, a CMDB export — into /tmp/from-authority.txt, then compare:

Read-only / Safeboth directions, separately— comm -23 shows lines only in the first file; comm -13 shows lines only in the second.

$ comm -23 /tmp/from-inventory.txt /tmp/from-authority.txt

web-a4.example.com

That is a ghost: the inventory has it, the authority does not.

Read-only / Safethe direction that matters more— Hosts the authority knows about that the inventory has never heard of.

$ comm -13 /tmp/from-inventory.txt /tmp/from-authority.txt

web-a5.example.com

That is a shadow host: it exists, and no automation touches it.

Both lists should normally be empty. When they are not, the report should go to the team that owns the affected service — which is available, because lesson 3 of this part put service_owner_team in the group.

Under the hood

--list-hosts prints a hosts (N): header line before the names, which is why the pipeline starts with tail -n +2. It writes to stdout, and warnings go to stderr, so a pattern that matches nothing produces an empty list rather than a corrupted one.

Two properties make this usable as a scheduled job. It connects to nothing, so it cannot disturb the fleet it is auditing and cannot fail because a host is down. And it is exact rather than approximate: it is Ansible’s own resolution of the inventory, not a parse of the files, so it accounts for ranges, aliases, multiple sources and merged directories the same way a real run would.

Where hosts appear under an alias with ansible_host, compare on the authority’s identifier rather than on the inventory name — otherwise every aliased host looks like a divergence in both directions at once.

Writing the decision down

The decision is worth about ten lines, at the root of the inventory, where the next operator will meet it before they meet the consequences:

# inventory/SOURCE-OF-TRUTH.md

Authoritative for host existence:
  - Cloud instances (eu-west, us-east): the cloud provider API.
      Group membership derives from the `environment` and `role` tags.
      A tag change IS an inventory change and is not reviewed. Tag edit
      permission is restricted to the platform team for this reason.
  - Physical hosts and appliances: this repository. Adding a machine is
      a pull request; there is no other path.

Reconciliation:
  - Job: ci/reconcile-inventory, daily 06:00 UTC.
  - Reports both directions to the owning team named in group_vars.
  - A non-empty result is a ticket, not an email.

Owner: platform-infra
Last reviewed: 2026-08-11

The specific content matters less than the fact that somebody had to write “a tag change is an inventory change and is not reviewed” and read it back. That sentence tends to prompt the follow-up question about who can edit tags, which is the control the model actually needs.

What happens if the authority is unavailable when a run starts?

This is where the choice stops being philosophical.

A Git-based inventory is a file on the controller, so a run works during an API outage. A provider-based inventory needs the provider to answer, and the failure is not always a clean one: depending on the plugin and its cache, a run may proceed against a partial fleet, which looks like a successful run that quietly skipped half the estate.

Two mitigations, and you want both. Cache the resolved inventory so a transient failure serves the last known good set rather than a truncated one. And assert on the host count at the start of any play that matters:

- name: Refuse to run against an implausibly small fleet
  ansible.builtin.assert:
    that:
      - ansible_play_hosts | length >= expected_minimum_hosts
    fail_msg: >-
      Inventory resolved to {{ ansible_play_hosts | length }} hosts,
      below the expected minimum of {{ expected_minimum_hosts }}.
      Refusing to proceed against a possibly partial inventory.
  run_once: true

A run against half the fleet is more dangerous than a run that fails, because it produces a fleet in two states and a recap that says everything went fine. The caching and staleness mechanics behind this belong to the dynamic inventory part; the guard belongs here, because it is a governance decision about what an incomplete answer is worth.