Fact caching and the risk of stale facts

The facts part of this course covers configuring a persistent cache and watching it work. This lesson is the performance-engineering view: where caching belongs in the sequence of changes, and why it is the only lever in this part whose failure mode is a wrong answer rather than a slow run.

Everything else here fails visibly. A saturated controller reports unreachable hosts. A free strategy produces a confusing transcript. A pipelining prerequisite that is not met produces sudo errors. Caching fails by succeeding — quickly, confidently, with a value that stopped being true two days ago.

Cache last

The order matters and it is routinely inverted, because caching is the change that sounds most like a performance feature.

1. Profile. Confirm gathering is the largest line.
2. Restrict gather_subset to what the audit proved you read.
3. Set gathering = smart if the playbook has multiple plays over overlapping hosts — this gathers once per host per run, with no staleness risk at all, because the facts are from this run.
4. Then, if gathering is still a significant cost across separate runs, cache.

Steps 2 and 3 remove work without introducing any claim about the past. Step 4 introduces one. Doing step 4 first means you are persisting facts you did not need to collect, and paying the staleness risk for the privilege.

Which plugins exist

$ ansible-doc -t cache -l

ansible.builtin.jsonfile JSON formatted files
ansible.builtin.memory   RAM backed, non persistent

Two. memory is the default and is not a cache in any useful sense — it lives for the duration of one run and is discarded. jsonfile writes one JSON file per host to a directory you choose, and is the only persistent option in core.

Everything else is in a collection, and this matters when you are pinning dependencies:

• community.general — redis, memcached, pickle, yaml
• community.mongodb — mongodb

redis is the usual choice where several controllers or CI runners share a cache, because jsonfile on a local directory gives each of them a separate, independently stale copy. That is a real operational difference: with jsonfile, a fact refreshed by your laptop run is not refreshed for the CI runner, so the same playbook can make different decisions depending on where it ran.

The configuration keys are worth knowing under their real names, because the setting name and the ini key differ:

$ ansible-config list | grep -A10 '^CACHE_PLUGIN'

CACHE_PLUGIN:            ini key: fact_caching            default: memory
CACHE_PLUGIN_CONNECTION: ini key: fact_caching_connection  default: null
CACHE_PLUGIN_PREFIX:     ini key: fact_caching_prefix      default: ansible_facts
CACHE_PLUGIN_TIMEOUT:    ini key: fact_caching_timeout     default: 86400

The default timeout is 86400 seconds — 24 hours. That is the number a cache runs with if nobody chose one, and it is far too long for several facts people routinely branch on.

--flush-cache and who is responsible for it

$ ansible-playbook --help | grep -A1 flush-cache

  --flush-cache         clear the fact cache for every host in inventory

Every host in inventory, not the hosts your --limit selects. That is usually what you want and it is worth knowing before you run it during an incident on a 2,000-host inventory — the next run for every one of those hosts pays full gathering price.

The operationally important point is not the flag. It is that cache invalidation is a responsibility somebody has to own, and Ansible does not own it. A cache entry expires on a timer; it does not expire because the host changed.

So the flush has to be wired into the workflows that change the things you cache:

# After any workflow that changes hardware, addressing or OS version:
#   - VM resize (CPU or memory)
#   - re-image / rebuild
#   - network reconfiguration or IP change
#   - distribution upgrade
ansible-playbook -i inventory/production resize.yml --limit web7
ansible-playbook -i inventory/production site.yml --flush-cache --limit web7

If the resize workflow is a different team’s pipeline, then the flush is a cross-team dependency, and it needs to be written down somewhere other than in one person’s memory. This is the part that makes fact caching an operational commitment rather than a configuration setting.

Deciding what is safe, by rate of change

The useful question is not “should we cache facts”. It is “how fast does each fact I depend on change, relative to my cache timeout”.

The pattern in the right-hand column: facts that describe the shape of the machine are safe; facts that describe its current capacity or addressing are not — because those are exactly the ones that change as a result of the operational work you do, and exactly the ones that get templated into configuration.

[defaults]
gathering = smart
fact_caching = ansible.builtin.jsonfile
fact_caching_connection = /var/lib/ansible/facts
# 4 hours, not the 86400s default. We branch on ansible_default_ipv4 in
# the proxy templates, and failover can re-address a host within a day.
# Any workflow that resizes or re-images MUST run with --flush-cache;
# see RUNBOOK-CACHE-INVALIDATION.
fact_caching_timeout = 14400

Knowledge check