Fact caching and stale truth · Ansible

Lesson 1 measured what gathering costs and the number was uncomfortable: 94% of a fifty-host run, before any work. Restricting gather_subset halved it. Fact caching removes it almost entirely.

It also introduces a hazard that nothing else in Ansible has. Every other failure mode in this course announces itself — a task fails, a host is unreachable, a conditional errors. A stale fact produces a green run that configured the wrong thing, and the run output contains no evidence that anything went wrong.

This lesson sets up caching, then breaks it deliberately so you can see what the failure looks like from the operator’s side.

Caching is always on; the question is which plugin

Upstream puts it precisely: fact caching is always active, and the default plugin is memory, which “only caches the data for the current execution of Ansible”. So the default is a cache that is discarded when the process exits — which is why you pay for gathering on every run.

Read-only / Safethe caching settings and their defaults— Reads the resolved values rather than assuming them. CACHE_PLUGIN_TIMEOUT is in seconds; 86400 is 24 hours.

$ ansible-config dump | grep -E 'CACHE_PLUGIN|GATHERING'

CACHE_PLUGIN(default) = memory
CACHE_PLUGIN_CONNECTION(default) = None
CACHE_PLUGIN_PREFIX(default) = ansible_facts
CACHE_PLUGIN_TIMEOUT(default) = 86400
DEFAULT_GATHERING(default) = implicit

Switching to a persistent plugin is three settings:

Configuration changeansible.cfg— Changes how the controller stores facts between runs. Does not touch any managed node, but it does change what subsequent plays believe about your fleet.

[defaults]
gathering = smart
fact_caching = jsonfile
fact_caching_connection = /var/cache/ansible/facts
fact_caching_timeout = 7200

gathering is the keyword that makes the cache useful, and its three values are worth knowing exactly:

Value	Behaviour
`implicit`	The default. Gather in every play unless `gather_facts: false`. The cache is written but never consulted for skipping.
`explicit`	Never gather unless a play sets `gather_facts: true`.
`smart`	Gather once per host per run, and skip gathering entirely for a host already in the cache.

smart is the one that pays for the cache. Without it, you write cache entries you never read.

Watching it work

Run the same play twice with a jsonfile cache and gathering: smart:

Read-only / Saferun one: gathers and writes the cache— First run against an empty cache. The Gathering Facts task appears and the play does two tasks worth of work.

$ ansible-playbook cache.yml

PLAY [cached] ******************************************************************

TASK [Gathering Facts] *********************************************************
ok: [localhost]

TASK [Show distribution] *******************************************************
ok: [localhost] => {
  "msg": "Ubuntu 26.04"
}

PLAY RECAP *********************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0

elapsed 0.58

Read-only / Saferun two: no gathering task at all— The same play against a populated cache. Gathering Facts is absent from the output entirely, ok drops from 2 to 1, and the run is roughly twice as fast.

$ ansible-playbook cache.yml

PLAY [cached] ******************************************************************

TASK [Show distribution] *******************************************************
ok: [localhost] => {
  "msg": "Ubuntu 26.04"
}

PLAY RECAP *********************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=0

elapsed 0.30

Look at what changed in the output: the Gathering Facts task is gone. Not skipped, not ok — absent. ok drops from 2 to 1.

That absence is the single most useful diagnostic signal in this lesson. If you are reading a run and there is no Gathering Facts line, every fact that play used came from disk, and the host was not asked anything.

Breaking it deliberately

Here is the scenario the break/fix for this part is built on. A play chooses a package manager from ansible_facts.os_family. The cache holds facts written before the host was rebuilt onto a different distribution.

Simulated by editing the cache entry directly and re-running:

Read-only / Safea completely green run that is completely wrong— The cache says RedHat. The host is Ubuntu. No gathering task ran, so nothing checked. The recap is clean.

$ ansible-playbook cache2.yml

PLAY [cached] ******************************************************************

TASK [Choose a package manager from the cached fact] ****************************
ok: [localhost] => {
  "msg": "Rocky / RedHat -> would use dnf"
}

PLAY RECAP *********************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=0

ok=1, failed=0, unreachable=0. By every signal Ansible offers, that run succeeded. It selected dnf for a Debian host.

--flush-cache discards the cache for the run and forces gathering:

Read-only / Safethe same play with --flush-cache— Discards cached facts at the start of the run and gathers fresh. The Gathering Facts task reappears and the decision changes.

$ ansible-playbook --flush-cache cache2.yml

PLAY [cached] ******************************************************************

TASK [Gathering Facts] *********************************************************
ok: [localhost]

TASK [Choose a package manager from the cached fact] ****************************
ok: [localhost] => {
  "msg": "Ubuntu / Debian -> would use apt"
}

PLAY RECAP *********************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0

Two runs of the same playbook, minutes apart, on an unchanged host, producing opposite decisions. Nothing in either recap distinguishes them. The only visible difference is the presence of the Gathering Facts task.

You cannot diff a cache entry against reality by reading the file

The obvious diagnostic — open the cache file and check — does not work the way you expect. On ansible-core 2.21 a jsonfile entry looks like this:

{
    "__payload__": "{\"ansible_distribution\": \"Ubuntu\", ...}"
}

The facts are a JSON string inside a wrapper object, not a plain dictionary, and the filename carries a prefix (s1_localhost rather than localhost). Upstream states the general rule directly: the cache “should not be interrogated external to the cache plugin itself”.

So do not grep the cache. Ask Ansible what it believes:

ansible db07 -m ansible.builtin.debug -a "var=ansible_facts.os_family"

With gathering: smart and a populated cache that returns the cached value. Add --flush-cache and it returns the live one. Running both and comparing is the actual diagnostic, and it works regardless of which cache plugin is configured.

Choosing a timeout

fact_caching_timeout defaults to 86400 seconds — 24 hours. That is a long time in an estate that rebuilds hosts.

The question to ask is not “how long do I want to cache” but “how fast does the fastest-changing fact I branch on change, and what happens if I act on the old value?”

What you branch on	Reasonable timeout	Why
`os_family`, `distribution`	Hours to a day	Changes only on rebuild or major upgrade — but that is exactly when it matters
`default_ipv4.address`	Minutes to an hour	Changes on re-IP, failover, DHCP lease
`mounts`, `devices`	Do not cache decisions on these	A play that checks free space from a cached fact is not checking free space
`memtotal_mb`, `processor_vcpus`	Hours	Changes on resize, which is a planned event you can hook

The row that matters most is the third. Anything you are checking as a precondition should not come from cache. A guard that refuses to deploy when a filesystem is over 90% full, reading a cached mounts fact, is a guard that tells you about disk usage from yesterday. That is worse than no guard, because it reads as protection.

Where a play needs both — cached identity facts for speed, live state facts for a guard — gather explicitly for the live part:

Read-only / Safelive facts for the thing that must be live— An explicit setup task takes module arguments rather than play keywords, and its results are current regardless of what the cache holds.

- name: Deploy with a disk-space precondition
hosts: appservers
# Identity facts arrive from cache under gathering: smart.
tasks:
  - name: Re-read the mount table, now, from the host
    ansible.builtin.setup:
      gather_subset:
        - '!all'
        - '!min'
        - hardware

  - name: Refuse to deploy onto a full filesystem
    ansible.builtin.assert:
      that:
        - item.size_available > 1073741824
      fail_msg: "{{ item.mount }} has under 1 GiB free"
    loop: "{{ ansible_facts.mounts }}"
    loop_control:
      label: "{{ item.mount }}"

Note the shape: setup as an explicit task takes gather_subset as a module argument. As a play keyword it configures the implicit gathering task; as a module argument it configures this call. Same name, two levels, and the task form is the one that lets you refresh part of the fact set mid-play.

`set_fact` and `cacheable`

set_fact normally creates a host variable that lives for the run. With cacheable: true it also writes a fact that the cache plugin will persist.

Upstream is unusually careful about what this does, and it is worth quoting because the behaviour surprises people: it “actually creates 2 copies of the variable, a normal set_fact host variable with high precedence and a lower ansible_fact one that is available for persistence via the facts cache plugin”.

Three consequences:

Precedence changes between runs. In the run that sets it, the value sits at the set_fact level — entry 19 of the precedence ladder from Part XIII, above almost everything. In the next run it arrives as a cached fact, at entry 11, below play vars, vars_files, role vars and more. A value that overrode your group_vars today is overridden by it tomorrow.

cacheable: true does not enable caching. It means the value will work with a cache if one is already configured. With the default memory plugin it persists for exactly as long as an ordinary set_fact would.

meta: clear_facts removes only one of the two copies. It clears the ansible_fact copy and leaves the host variable, which is a genuinely confusing state to debug.

The honest guidance: cacheable: true is for genuinely derived, genuinely expensive, genuinely stable values — the result of an inventory API call, a computed cluster topology. For anything else the precedence shift between run one and run two is a bug waiting for a quiet afternoon.

Under the hood

The cache is keyed per host and stores the fact dictionary as the gathering task produced it. Under gathering: smart, before inserting the implicit setup task Ansible checks whether the cache holds a non-expired entry for each host; hosts with a hit are excluded from the gathering task, and if every host hits, the task is not inserted at all.

That is why the task disappears rather than showing as skipped.

Expiry is evaluated at read time against fact_caching_timeout, so an entry does not get cleaned up when it expires — it is simply ignored and rewritten on the next gather. A jsonfile cache directory therefore accumulates entries for hosts that left the inventory years ago. Harmless for correctness, and worth knowing when the directory has 40,000 files in it.

The plugin interface is deliberately opaque, which is the design decision behind the __payload__ wrapper you saw earlier. Upstream asks that playbooks work identically whether the cache hits or misses, and that nothing outside the plugin reads its storage — because the storage format is the plugin’s business and has changed before.

What happens if two runs use the same cache with different gather_subset settings?

The second run overwrites the first run’s facts with its own, narrower set, and a third play that expected the wider set finds facts missing.

This is a real and common footgun in an estate with several playbooks sharing one cache directory. A nightly compliance play tuned to ['!all', '!min', 'distribution'] writes a cache entry containing eight facts. A deployment play the next morning runs under gathering: smart, gets a cache hit, skips gathering, and then fails on ansible_facts.default_ipv4.address — which nothing gathered.

The failure looks like a bug in the deployment play. It is a consequence of what a different playbook wrote overnight.

Two defences:

Separate cache connections for playbooks with materially different fact needs. fact_caching_connection is just a path; give the compliance run its own.
Gather what you use, explicitly, in a play that has real requirements. An explicit setup task with the subset the play needs costs one round trip and removes the dependency on what someone else’s playbook left in the cache.

Knowledge check

Why this matters

Fact caching is the clearest trade in the course. You give up currency and you get speed, and on a large fleet the speed is not a nicety — without a cache, an hourly run over thousands of hosts spends most of its budget re-asking questions whose answers did not change.

What makes it dangerous is that the cost of the trade is invisible in the run output. Every other mistake in Ansible eventually shows you a red line. A stale fact gives you a clean recap and a wrong outcome, and the operator reading that recap has no way to tell.

So the discipline is: know which facts your plays branch on, know how fast each of them can change, and never evaluate a safety precondition against a cached value. When a run surprises you, check whether the Gathering Facts task was there before you check anything else.