Inventory caching and staleness

An inventory plugin makes an API call. API calls cost time, cost money on some providers, and are rate limited on all of them. Ansible’s answer is a cache: keep the last answer, reuse it until it expires.

That is a sensible engineering decision and it introduces a genuinely new operational object into your estate — a host list that is a picture of the fleet at some point in the past, and whose age you cannot see in the output.

Both directions of that are dangerous, and they are dangerous in different ways, which is the point this lesson is built around.

The four options, and where they come from

Every inventory plugin that supports caching gets the same options, because they come from a single documentation fragment that plugins include rather than each defining their own. That is why the AWS plugin, the Azure plugin and a plugin you write yourself all spell it the same way.

No core inventory plugin includes that fragment — constructed, yaml and ini read local files and have nothing to cache — so the place to read the authoritative defaults is the fragment source that ships with ansible-core:

$ sed -n '12,48p' $(python3 -c 'import ansible,os;print(os.path.dirname(ansible.__file__))')/plugins/doc_fragments/inventory_cache.py

options:
cache:
  description:
    - Toggle to enable/disable the caching of the inventory's source data, requires a cache plugin setup to work.
  type: bool
  default: no
  env:
    - name: ANSIBLE_INVENTORY_CACHE
  ini:
    - section: inventory
      key: cache
cache_plugin:
  description:
    - Cache plugin to use for the inventory's source data.
  type: str
  default: memory
  env:
    - name: ANSIBLE_CACHE_PLUGIN
    - name: ANSIBLE_INVENTORY_CACHE_PLUGIN
  ini:
    - section: defaults
      key: fact_caching
    - section: inventory
      key: cache_plugin
cache_timeout:
  description:
    - Cache duration in seconds.
  default: 3600
  type: int
  env:
    - name: ANSIBLE_CACHE_PLUGIN_TIMEOUT
    - name: ANSIBLE_INVENTORY_CACHE_TIMEOUT
  ini:
    - section: defaults
      key: fact_caching_timeout
    - section: inventory
      key: cache_timeout

Two details in that output are worth noticing now, because they cause confusion later. Each option has two environment variables and two ansible.cfg keys, and one of each pair is the fact-caching name — ANSIBLE_CACHE_PLUGIN, [defaults] fact_caching. Setting those affects inventory caching and fact caching together. The ANSIBLE_INVENTORY_CACHE_* and [inventory] forms affect inventory caching only, and are the ones to prefer when you mean one and not the other.

memory is not a cache in the useful sense

The default cache_plugin is memory, and memory means this process. It disappears when the command exits. It is genuinely useful — a single playbook run that references the inventory several times hits the API once — but between two ansible-playbook invocations it does nothing at all.

Only two cache plugins ship in core:

$ ansible-doc -t cache -l

ansible.builtin.jsonfile JSON formatted files
ansible.builtin.memory   RAM backed, non persistent

So a cache that survives between runs means jsonfile unless you have installed a collection. jsonfile writes one file per source into the directory given by cache_connection.

plugin: example.provider.source
cache: true
cache_plugin: jsonfile
cache_connection: /var/cache/ansible/inventory
cache_timeout: 900

What a cache hit looks like from outside

Nothing. That is the entire problem.

Here is a source whose backing data gains a host between two runs, with cache: true and a jsonfile cache:

$ ansible-inventory -i demo.yml --graph

@all:
|--@ungrouped:
|--@web:
|  |--web01.example.com
|  |--web02.example.com
|--@db:
|  |--db01.example.com

The source now contains a fourth host, web03.example.com. Run it again, unchanged:

$ ansible-inventory -i demo.yml --graph

@all:
|--@ungrouped:
|--@web:
|  |--web01.example.com
|  |--web02.example.com
|--@db:
|  |--db01.example.com

$ ansible-inventory -i demo.yml --graph --flush-cache

@all:
|--@ungrouped:
|--@web:
|  |--web01.example.com
|  |--web02.example.com
|  |--web03.example.com
|--@db:
|  |--db01.example.com

Read the two graphs again. They are the same command producing different answers, and there is nothing in the first one that tells you it is old. An inventory cache has no visible timestamp in normal output. The only way to know how old a host list is, is to know how the cache is configured.

Both directions are real, and they are not symmetrical

The instinct is “stale is bad, fresh is good”. That is half right, and the half that is wrong is the half that causes outages.

Stale: you miss hosts, and you target ghosts

A cached list that predates a scale-up misses the new instances. A security patch run reports complete success across every host it knew about, and three machines that came up forty minutes ago are unpatched. The recap is green. Nothing indicates a gap.

A cached list that predates a scale-down contains machines that no longer exist. Those show as unreachable, which is at least loud — but a run configured to tolerate unreachable hosts absorbs them silently, and now “3 unreachable” is a number people have learned to ignore.

Fresh: your blast radius changes without a diff

Now the other direction, which people do not expect.

A run with --flush-cache sees the fleet as it is right now, which includes anything created in the last five minutes by anybody. If your grouping is opt-out — a host is managed unless tagged otherwise — then a freshly queried inventory is precisely how a machine nobody told you about ends up in a play.

Staleness gives you a small, bounded window of not knowing. Freshness gives you an unbounded set of hosts you have never reviewed. Neither is safe on its own; what makes either safe is knowing which one you are getting and verifying the host list before the run.

Flushing, and the flag that is easy to misread

--flush-cache exists on both ansible-playbook and ansible-inventory, and it does two related things:

1. It discards the existing cache for the inventory sources in play, so the plugin re-queries the provider.
2. It repopulates the cache with the new answer, so the next run without the flag sees the fresher list rather than the old one.
3. On ansible-playbook it also clears the fact cache for the hosts in the run, which is a separate cache with a separate purpose - if you are using fact caching to avoid re-gathering, this flag throws that away too.

Point three is the one that surprises people, and it is why --flush-cache should not be pinned into every scheduled command “to be safe”. A nightly job that flushes on every run re-queries the provider API and re-gathers facts across the whole fleet, which is a large amount of work to buy a freshness guarantee you could have got by setting cache_timeout sensibly.

Knowledge check