Ad-hoc as a fleet inspection instrument

This is the lesson where ad-hoc commands are unambiguously the right tool, and it is worth being as enthusiastic about it as the rest of this part is cautious.

An estate-wide question that would take an hour of SSH and note-taking takes four seconds. “Which of these 200 hosts are still on the old kernel?” “Which ones have the wrong resolver?” “Is that package the version we think it is anywhere?” These are read-only, they are safe to run in the middle of an incident, and the output is evidence you can put in a ticket.

The whole argument of this part is that ad-hoc inspection is fine forever and ad-hoc change is a debt. This lesson is the first half.

Start with reachability

Before any inspection tells you something about the estate, establish which parts of the estate answered.

$ ansible production -i inventory.ini -m ansible.builtin.ping

web01.example.com | SUCCESS => {
  "changed": false,
  "ping": "pong"
}
db01.example.com | UNREACHABLE! => {
  "changed": false,
  "msg": "Task failed: Failed to connect to the host via ssh: ssh: connect to host 198.51.100.21 port 22: Connection timed out",
  "unreachable": true
}

ping here is not ICMP. It is a Python module that runs on the target and returns the string pong, so a success proves three things at once: the network path works, SSH authenticated, and there is a usable Python interpreter. A failure at any of those layers looks different, which makes it a better first probe than ssh host true.

Facts: the broadest question you can ask

ansible.builtin.setup is the module that fact gathering uses, and you can call it directly. Unfiltered it returns several hundred keys per host, which is unreadable and slow. Filter it.

$ ansible production -i inventory.ini -m ansible.builtin.setup -a "filter=ansible_distribution*"

web01.example.com | SUCCESS => {
  "ansible_facts": {
      "ansible_distribution": "Ubuntu",
      "ansible_distribution_file_parsed": true,
      "ansible_distribution_file_path": "/etc/os-release",
      "ansible_distribution_file_variety": "Debian",
      "ansible_distribution_major_version": "24",
      "ansible_distribution_release": "noble",
      "ansible_distribution_version": "24.04"
  },
  "changed": false
}

filter is a list of fnmatch patterns — shell-style globs, not regular expressions. It has accepted a list since Ansible 2.11 and still accepts a bare string as a single pattern. Two named facts, no globbing:

$ ansible production -i inventory.ini -m ansible.builtin.setup -a '{"filter": ["ansible_kernel", "ansible_memtotal_mb"]}'

web01.example.com | SUCCESS => {
  "ansible_facts": {
      "ansible_kernel": "6.8.0-51-generic",
      "ansible_memtotal_mb": 15990
  },
  "changed": false
}

filter is not the cheap knob — gather_subset is

This is the distinction that matters on a large fleet, and it is easy to get backwards.

filter runs after collection. Ansible gathers everything and then discards the keys you did not ask for. The transfer is smaller; the work on the target is identical.

gather_subset changes what is collected. Excluding the hardware subset means the module never enumerates block devices, never reads /proc/partitions, and never waits on a hung disk.

$ ansible production -i inventory.ini -m ansible.builtin.setup -a "gather_subset=!all,!min,virtual"

web01.example.com | SUCCESS => {
  "ansible_facts": {
      "ansible_virtualization_role": "guest",
      "ansible_virtualization_tech_guest": [
          "kvm"
      ],
      "ansible_virtualization_type": "kvm",
      "gather_subset": [
          "!all",
          "!min",
          "virtual"
      ],
      "module_setup": true
  },
  "changed": false
}

The subsets are named in ansible-doc ansible.builtin.setup and include hardware, network, virtual, mounts, devices, distribution, kernel, python, service_mgr, pkg_mgr and about forty more. A leading ! excludes.

On five hosts this is a micro-optimisation. On two thousand, with a host whose SAN mount is wedged, the difference is between an inspection that returns in six seconds and one that hangs on the mounts collection until the connection times out.

Reading a file across the estate

The obvious approach is -m command -a "cat /etc/resolv.conf". Do not. It reports changed on every host, it returns an unstructured blob, and it fails differently depending on the target’s shell.

slurp is the purpose-built module:

$ ansible production -i inventory.ini -m ansible.builtin.slurp -a "src=/etc/resolv.conf"

web01.example.com | SUCCESS => {
  "changed": false,
  "content": "bmFtZXNlcnZlciAxOTIuMC4yLjUzCg==",
  "encoding": "base64",
  "source": "/etc/resolv.conf"
}

The base64 is not an obstacle; it is what makes the return value safe to transport regardless of what the file contains. Decode it where you need it:

$ jq -r '.content' ./evidence/web01.example.com | base64 -d

nameserver 192.0.2.53

For a question about a file’s metadata rather than its contents — “does this exist”, “who owns it”, “has it changed” — stat is cheaper and returns structured fields including a checksum:

$ ansible production -i inventory.ini -m ansible.builtin.stat -a "path=/etc/nginx/nginx.conf"

web01.example.com | SUCCESS => {
  "changed": false,
  "stat": {
      "checksum": "0f1e2d3c4b5a69788796a5b4c3d2e1f00f1e2d3c",
      "exists": true,
      "gr_name": "root",
      "mode": "0644",
      "mtime": 1783030738.4909694,
      "path": "/etc/nginx/nginx.conf",
      "pw_name": "root",
      "size": 1447
  }
}

The checksum answers a question nothing else answers cheaply: are these 200 files identical? Collect them, sort, count the distinct values. One value means the fleet is consistent. Three means you have a drift problem and now know its shape.

The rest of the read-only toolkit

Two cautions on that table. package_facts on a large RPM estate is genuinely expensive — it enumerates the whole package database — so scope it with --limit before you run it on two thousand hosts. And uri is read-only only if you leave method at its default of GET; the same module will happily POST or DELETE.

Capture it as evidence

Terminal scrollback is not evidence. It is not shareable, it is not timestamped, and it disappears.

$ ansible production -i inventory.ini -m ansible.builtin.setup -a "filter=ansible_kernel" --tree ./evidence

$ jq -r '.ansible_facts.ansible_kernel' ./evidence/* | sort | uniq -c | sort -rn

     37 6.8.0-51-generic
    2 6.8.0-45-generic
    1 5.15.0-119-generic

Three lines, and the third one is the finding. That host is two LTS kernels behind and nobody knew.

This is the pattern worth internalising: run the inspection with --tree, process the JSON locally, put the summary in the ticket. It takes ten seconds longer than reading the screen and produces something a colleague can check.

Knowledge check