Inventory performance and truth at scale

At forty hosts, inventory parsing is instantaneous and correct, and you never think about it. At five thousand hosts it is neither, and both problems have consequences that show up somewhere else entirely.

The performance problem is easy to describe: a dynamic inventory source that queries a CMDB or a cloud API for five thousand hosts takes real time, and that time is paid before every single Ansible command, not once per session. The truth problem is harder: an inventory that is a day out of date describes a fleet that no longer exists, and a run against it is a run with the wrong blast radius.

Where the time goes, and when

Inventory is parsed at the start of every invocation: every ansible-playbook, every ad-hoc ansible, every ansible-inventory. There is no daemon and no session; the process starts, builds the whole inventory, and exits.

time ansible-inventory -i inventory/ --list > /dev/null
time ansible-inventory -i inventory/ --list > /dev/null
time ansible-inventory -i inventory/ --list > /dev/null

Interpreting the result:

The important observation is that the cost falls hardest on the cheap operations. A four-hour fleet run does not care about a thirty-second parse. But “let me just check whether that host is up” becomes a thirty-second wait, and after the third time nobody checks. Slow inventory quietly removes ad-hoc inspection from a team’s habits, and ad-hoc inspection is the thing that catches problems before they become runs.

Caching a slow source

Inventory plugins that talk to an API can cache their results. Caching is configured on the plugin, in the plugin’s own config file, and typically looks like this:

plugin: example.cmdb.inventory
strict: true

cache: true
cache_plugin: jsonfile
cache_connection: /var/cache/ansible/inventory
cache_timeout: 3600

The trade is stated most usefully as a question about blast radius: a cached inventory is a description of the fleet from up to cache_timeout seconds ago. Everything downstream inherits that staleness — which hosts a pattern matches, how many hosts a wave contains, what --list-hosts reports.

# rebuild inventory from source, ignoring any cache
ansible-inventory -i inventory/ --list --flush-cache --output inventory-snapshot.json

# what the fleet looked like when the change was approved
jq '._meta.hostvars | keys | length' inventory-snapshot.json

A snapshot attached to the change record answers the question that comes up in every post-incident review of a fleet change: which hosts were in scope when this was approved?

Per-wave limit files, generated not typed

The previous lesson used fixed partition files for resumability. The same artefacts serve a second purpose: they are the record of what a wave meant.

set -euo pipefail

for wave in wave0_tooling wave1_canary wave2_internal wave3_domain_a; do
ansible-inventory -i inventory/ --graph "$wave" \
  | sed -n 's/^  *|--\([^@].*\)$/\1/p' \
  | sort -u > "limits/${wave}.hosts"
printf '%-20s %s hosts\n' "$wave" "$(wc -l < "limits/${wave}.hosts")"
done

Committing those files makes wave membership reviewable in a pull request, which is a materially different thing from a --limit someone types at 02:00.

Truth: reconciling in both directions

Inventory accuracy is two separate problems with two separate symptoms.

Hosts in inventory that no longer exist. Every run reports them unreachable. Individually harmless; collectively corrosive. A fleet run that always shows forty unreachable hosts has an unreachable noise floor of forty, and the night one real host goes unreachable the number is forty-one and nobody notices. This is precisely the mechanism by which an inaccurate changed destroys drift detection, applied to a different counter.

Hosts that exist but are not in inventory. These are worse, because they produce no signal at all. They are not patched, not audited, not converged, and not counted. They are the hosts an incident finds.

set -euo pipefail

ansible-inventory -i inventory/ --list \
| jq -r '._meta.hostvars | keys[]' | sort > /tmp/in-inventory.txt

# from monitoring, the hypervisor, DNS - anything that is not the inventory
curl -sf https://monitoring.example.com/api/hosts \
| jq -r '.[].name' | sort > /tmp/in-reality.txt

echo "== in inventory, not in reality (unreachable noise) =="
comm -23 /tmp/in-inventory.txt /tmp/in-reality.txt

echo "== in reality, not in inventory (unmanaged hosts) =="
comm -13 /tmp/in-inventory.txt /tmp/in-reality.txt

An unparseable source is only a warning

This one deserves its own section because the default is surprising and the consequence is a wrong blast radius.

$ ansible-playbook -i inventory/hosts.ini -i inventory/missing.ini site.yml --list-hosts

[WARNING]: Unable to parse /path/to/inventory/missing.ini as an inventory source

playbook: site.yml

play #1 (web): web	TAGS: []
  pattern: ['web']
  hosts (10):
    web08
    web09
    ...

INVENTORY_UNPARSED_IS_FAILED and INVENTORY_ANY_UNPARSED_IS_FAILED both default to false on 2.21.3. So a typo in a path, a dynamic inventory plugin whose credentials expired, or a YAML file with a syntax error produces a warning scrolling past at the top of the output — and then the run proceeds against the hosts that did parse.

At fleet scale that is the ideal shape of a silent disaster: half the inventory missing, no error, a clean recap, and a change applied to a subset nobody chose.

[defaults]
inventory_unparsed_is_failed = true
any_unparsed_is_failed = true

With any_unparsed_is_failed enabled, the same run produces [ERROR]: Completely failed to parse inventory source ... instead of a warning — confirmed on 2.21.3.

Knowledge check