delegate_facts: whose facts are these? · Ansible

A task that produces a fact and a task that produces a change are treated very differently by this course, and delegation is where the distinction earns its keep.

A delegated change is loud: something happened on another machine, and if it was the wrong machine you find out. A delegated fact is silent. It lands in the run’s variable store under some host’s name, and nothing checks that the name matches the machine the measurement came from. A run that misfiles every fact it gathers reports ok on every task.

The rule is one sentence, and the reason it surprises people is that it is the opposite of what “delegate” suggests.

The default

Facts produced during a delegated task are stored against inventory_hostname — the original host — not against the host the task ran on.

Demonstrated, with set_fact standing in for anything that produces a fact, on two app servers delegating to lb1:

Read-only / Safewhere a delegated fact lands— Run on ansible-core 2.21.3. Every host uses ansible_connection=local and every task is set_fact or debug, so nothing outside the controller process is touched.

$ ansible-playbook -i deleg.ini delegfacts.yml

TASK [Set a fact while delegated, WITHOUT delegate_facts] **********************
ok: [web1 -> lb1]
ok: [web2 -> lb1]

TASK [Where did it land - original host] ***************************************
ok: [web1] => {
  "msg": "on web1: probe_source=gathered-while-delegated"
}
ok: [web2] => {
  "msg": "on web2: probe_source=gathered-while-delegated"
}

TASK [Where did it land - lb1] *************************************************
ok: [web1] => {
  "msg": "hostvars[lb1].probe_source=UNSET"
}

The task ran on lb1. The fact is on web1 and web2. lb1 has nothing.

Now the same shape with delegate_facts: true:

Read-only / Safedelegate_facts sends it to the delegate instead— Same run, later tasks. Note the word instead: the fact does not land in both places.

$ ansible-playbook -i deleg.ini delegfacts.yml

TASK [Set a fact while delegated, WITH delegate_facts] *************************
ok: [web1 -> lb1]
ok: [web2 -> lb1]

TASK [Check both again] ********************************************************
ok: [web1] => {
  "msg": "self.probe2=UNSET | lb1.probe2=landed-on-delegate"
}
ok: [web2] => {
  "msg": "self.probe2=UNSET | lb1.probe2=landed-on-delegate"
}

delegate_facts: true moves the destination. It does not duplicate it. After that task, web1 does not have probe2 at all — a detail that matters if a later task on web1 was written expecting it.

The keyword documentation says the same thing in one line:

Read-only / Safethe keyword, from the installed core— ansible-doc reads the local installation, so this is the definition your controller is actually using.

$ ansible-doc -t keyword delegate_facts

delegate_facts:
applies_to:
- Role
- Block
- Task
- Handler
description: Boolean that allows you to apply facts to a delegated host instead
  of inventory_hostname.
priority: 0
template: explicit
type: bool

Note applies_to: it is not a play-level keyword. You set it on the task, block, role or handler that is doing the delegating.

Why the default is the right one

It looks like a trap, and it is one — but reversing it would be worse.

Most delegated tasks exist to learn something about the current host from somewhere else. Query the monitoring API for this host’s alert state. Ask the load balancer whether this host is still receiving traffic. Look up this host’s record in the CMDB. In every one of those, the answer belongs to the app server even though the question was asked from elsewhere:

- name: Ask the balancer whether this host is still serving
  ansible.builtin.uri:
    url: "https://lb.example.com/api/backends/{{ inventory_hostname }}"
    headers:
      Authorization: "Bearer {{ lb_api_token }}"
  register: pool_state
  delegate_to: "{{ app_loadbalancer }}"
  no_log: true

pool_state is web1’s pool state. Storing it against lb1 would be wrong, and would also mean every host in the play overwrote the previous one’s answer.

The default serves the common case. delegate_facts: true is for the uncommon one, and the uncommon one is worth naming precisely: you want facts about the delegate itself.

Gathering facts about a host that is not in the play

This is the legitimate use, and it is genuinely useful.

Your play targets app servers. You want to make a decision based on the load balancer’s operating system, its available memory, or which service_mgr it runs — and the load balancer is not in the play, so nothing has ever gathered facts for it.

Read-only / Safegather facts for a host outside the play— The setup module reads system information. It changes nothing, which is why gathering is READ-ONLY even against production.

- name: Gather facts about the load balancer
ansible.builtin.setup:
  gather_subset:
    - '!all'
    - '!min'
    - service_mgr
    - distribution
delegate_to: "{{ app_loadbalancer }}"
delegate_facts: true
run_once: true

- name: Now this is answerable
ansible.builtin.debug:
  msg: >-
    Balancer {{ app_loadbalancer }} runs
    {{ hostvars[app_loadbalancer]['ansible_distribution'] }}
    under {{ hostvars[app_loadbalancer]['ansible_service_mgr'] }}

Three things in that task are deliberate:

delegate_facts: true puts the facts on the balancer, where they belong and where hostvars[...] can find them.
gather_subset restricts collection to the two subsets actually needed. Full gathering against a delegate is a real cost paid once per host in the play — see the next paragraph — and there is no reason to collect network interfaces and mount tables to answer a question about the service manager.
run_once: true because otherwise this gathers the balancer’s facts once for every host in the play. Fifty app servers means fifty setup runs against one proxy, which is a self-inflicted load spike on the machine you are trying to be careful with. run_once has semantics of its own that lesson 5 covers in detail; here it is doing the obvious job.

What the failure actually looks like

The break/fix for this lesson is a real shape, and it is instructive because the first diagnosis is always wrong.

A team writes a play over web that gathers facts from the balancer to decide how to reload it, and forgets delegate_facts. The symptom reported is: “the playbook thinks our load balancer is Ubuntu, but it is Rocky Linux.”

The obvious diagnoses — a wrong inventory entry, a stale fact cache, an ansible_host pointing at the wrong machine — are all false. Nothing is misconfigured. The facts describing the balancer were gathered correctly from the balancer, and then filed under web1, web2, web3, silently overwriting each app server’s real facts with the proxy’s.

This corrupts facts you did not intend to touch

The damage is not limited to the fact you wanted.

ansible.builtin.setup returns the whole fact set for the machine it ran on. Without delegate_facts, that entire set is written into the original host’s namespace. ansible_distribution, ansible_default_ipv4, ansible_memtotal_mb, ansible_processor_count, ansible_mounts — every one of them now describes the proxy while claiming to describe the app server.

Any later task in that play which branches on a fact is now branching on the wrong machine’s data. Sizing a JVM heap from ansible_memtotal_mb. Choosing a package name from ansible_os_family. Writing a config file containing ansible_default_ipv4.address. All of them produce plausible output and none of it is right, and because the values came from a real machine there is nothing obviously malformed to notice in review.

If you find a missing delegate_facts in a play that has been running, the question is not only “when do we fix it” but “what did the runs after that task configure, and on which hosts”. Treat it as a data-integrity incident, not a syntax fix.

The diagnostic is direct, once you suspect it:

Read-only / Safecompare what a host claims against what it is— Two read-only fact queries. Run them against one host, not the group, while you are working out what happened.

# What does the run believe about web1?
ansible -i inventory/production web1 -m ansible.builtin.setup \
-a 'gather_subset=!all,!min,distribution' 2>&1 | grep -E 'distribution|hostname'

# What does the run believe about the balancer?
ansible -i inventory/production lb1 -m ansible.builtin.setup \
-a 'gather_subset=!all,!min,distribution' 2>&1 | grep -E 'distribution|hostname'

If a direct setup against web1 disagrees with what the playbook reported for web1, something in the play wrote facts into web1 from elsewhere. Search the repository for delegate_to on tasks that produce facts: setup, set_fact, and any module used with register where the result is later promoted to a fact.

Under the hood

register and facts are different mechanisms and only one of them is affected by delegate_facts.

register stores a task result in a per-host variable. It is always stored against inventory_hostname and delegate_facts does not move it. A delegated task with register: result puts the delegate’s result into the original host’s result, which is exactly what you want for the “ask the balancer about this host” pattern and is not configurable.

Facts — from setup, from set_fact, or from set_fact with cacheable: true — go through the fact store, and that is what delegate_facts redirects.

The distinction has a practical edge. set_fact with cacheable: true writes into the fact cache, so a delegated, cacheable set_fact without delegate_facts persists the misfiling to disk and survives the run. The next play that reads that host from cache sees the wrong data, with no delegated task anywhere in sight to explain it.

If you are unsure which mechanism a value came from, register results have the structure of a module return — changed, failed, stdout, rc — and facts do not.

What happens if you set delegate_facts: true without delegate_to?

Nothing useful. The keyword redirects facts to the delegated host, and with no delegation there is no delegate, so the facts land where they would have anyway — on inventory_hostname.

It produces no error and no warning, which makes it a quiet form of misleading code: a reader sees delegate_facts: true and reasonably assumes something is being redirected somewhere. This happens most often after a refactor removes a delegate_to and leaves its companion behind.

The related and more dangerous version is the reverse: a delegate_to added later to a task that gathers facts, without the author noticing that delegate_facts was needed. That one has no symptom at all until a downstream task uses the corrupted fact.

Both are worth a grep in any repository that uses delegation:

grep -rn -B4 'delegate_facts' roles/ playbooks/ | grep -c delegate_to

If the count of delegate_to lines near delegate_facts does not match the count of delegate_facts lines, read every one of them.

The third option is usually the best one

Before reaching for delegate_facts, ask whether the delegated task needs to branch on the delegate’s facts at all.

A drain step does not. It opens a socket, sends a command, and reports. Whether the proxy runs systemd or OpenRC is irrelevant to it, and a task that does not need to know is a task that cannot be wrong about it.

The cases that genuinely need the delegate’s facts are narrower than they first appear: choosing a package name or service name for a proxy you are also managing, sizing something to the delegate’s hardware, or writing a config file whose content depends on the delegate’s addresses. Every one of those is a sign the delegate is a managed host in its own right — and the cleaner design is a separate play that targets the proxies directly, where facts are gathered normally and none of this applies.

delegate_facts is a good tool for reading a machine you do not manage. It is a weak substitute for putting a machine you do manage into a play of its own.

Knowledge check

Why this matters

Fact corruption is the worst kind of automation defect because it is invisible at the point of failure and load-bearing everywhere else. A change applied to the wrong host announces itself. A fact filed under the wrong host waits, and then quietly sizes a service to another machine’s memory, or picks a package name from another machine’s operating system, and every affected run reports success.

delegate_facts is a one-line keyword protecting against that, and the default is not the safe-looking one — it is the one that suits the common pattern of asking a remote system about the host you are working on. That makes it your job to notice which of the two patterns you are writing.

The reviewing question is short enough to use every time: this task ran somewhere else — whose measurement is this?