Checksums, GPG signatures and what they prove · Ansible

Lesson 5 read a collection’s own claims about itself. This lesson checks a much narrower and much harder thing: that the bytes on your controller are the bytes somebody published.

That is a real and useful property. It is also, routinely, mistaken for a different and far larger one. A verified collection is not a safe collection. It is an unaltered one — which means the publisher’s code, exactly as the publisher wrote it, including anything they wrote badly or on purpose.

The two checksum files

Every collection installed from a Galaxy artefact carries two extra files that were not in the source tree:

Read-only / Safewhat an installed collection has that a git clone does not— Lists an installed collection directory. MANIFEST.json and FILES.json are added by the build, not written by hand.

$ ls -a collections/ansible_collections/example_corp/platform

.
..
FILES.json
MANIFEST.json
README.md
docs
meta
plugins
roles

FILES.json holds a SHA-256 checksum for every file in the collection. MANIFEST.json holds the collection metadata plus a checksum of FILES.json. That is a two-level chain: verify MANIFEST.json, and it vouches for FILES.json, which vouches for every file.

This is also the answer to the * version from lesson 3. A collection checked out from git has neither file, so there is nothing to verify against — which is the practical reason a production controller should install artefacts rather than clone repositories.

Verifying, and watching it catch something

Verification against the local manifest, with no network:

Read-only / Safeinternally consistent— Verifies an installed collection against its own manifest without contacting any server. Read-only: it computes checksums and compares them.

$ ansible-galaxy collection verify example_corp.platform -p ./collections --offline

Verifying 'example_corp.platform:1.0.0'.
Installed collection found at '/home/opsuser/project/collections/ansible_collections/example_corp/platform'
MANIFEST.json hash: b7a3fc1e750306f07c857feb1f8fb374656e158fdf7101656503c9b03ba146df
Successfully verified that checksums for 'example_corp.platform:1.0.0' are internally consistent with its manifest.

Read the wording carefully: internally consistent with its manifest. That phrasing is doing real work, and the next command shows why it is not a stronger claim.

Now append one line to a file in the installed collection — the sort of thing a “quick local fix” looks like — and verify again:

Read-only / Safetampering is detected, by name— Verifies the same collection after a single line was appended to one file. Exit status is 1 and the modified file is named.

$ ansible-galaxy collection verify example_corp.platform -p ./collections --offline; echo "rc=$?"

Verifying 'example_corp.platform:1.0.0'.
Installed collection found at '/home/opsuser/project/collections/ansible_collections/example_corp/platform'
MANIFEST.json hash: b7a3fc1e750306f07c857feb1f8fb374656e158fdf7101656503c9b03ba146df
Collection example_corp.platform contains modified content in the following files:
  README.md
rc=1

Named file, non-zero exit. That is a check you can put in CI and in a pre-change gate.

`--offline` versus online

Drop --offline and the command downloads the published artefact and compares against that:

Read-only / Safeverified against what the server actually serves— Verifies an installed collection by downloading the published artefact and comparing checksums against it. Requires network access to the Galaxy server.

$ ansible-galaxy collection verify community.docker -p ./collections

Verifying 'community.docker:4.7.0'.
Installed collection found at '/home/opsuser/project/collections/ansible_collections/community/docker'
Downloading https://galaxy.ansible.com/api/v3/plugin/ansible/content/published/collections/artifacts/community-docker-4.7.0.tar.gz to /home/opsuser/.ansible/tmp/ansible-local-162019a7u02r88/tmph1rz50he/community-docker-4.7.0-e9vko67e
MANIFEST.json hash: bedb151342e01b6945da014d533ef46128d1e82542722ddecc1f751a7b8fd2f6
Successfully verified that checksums for 'community.docker:4.7.0' match the remote collection.

The closing line differs: match the remote collection, not internally consistent with its manifest. Two different claims:

Mode	Compares	Catches	Misses
`--offline`	Files on disk against the local `MANIFEST.json`/`FILES.json`	Local modification after install	A manifest that was itself replaced, and anything wrong upstream
online	Files on disk against the freshly downloaded artefact	Local modification, and a manifest replaced along with the files	Anything wrong upstream

The --offline gap is real but narrow. An attacker who can rewrite files in your collections directory can also rewrite MANIFEST.json and FILES.json to match. Offline verification catches the careless edit; it does not catch the deliberate one.

Both miss the same larger thing, which is the point of the whole lesson.

Verification proves provenance, not intent

Say it in full, because the shortened version is where people go wrong:

“ansible-galaxy collection verify succeeded” means the files on your controller are the files the publisher built. It does not mean the publisher is trustworthy, that the code is correct, that it has been reviewed, or that it does what its documentation says.

A backdoor introduced by the publisher verifies perfectly. A signed release from a maintainer whose account was compromised verifies perfectly. A collection that stores credentials in world-readable files verifies perfectly.

Verification is one control among several, and its job is narrow: detecting substitution and modification in transit and at rest. Whether the thing being protected deserves protecting is lesson 5’s question, and no cryptography answers it.

GPG signatures

Galaxy servers can serve detached GPG signatures for a collection’s MANIFEST.json. ansible-galaxy will check them at install time if you give it a keyring.

Configuration changeinstall with signature requirements

ansible-galaxy collection install -r requirements.yml \
-p ./collections \
--keyring ~/.ansible/galaxy-keyring.kbx \
--required-valid-signature-count +all

The options that matter:

--keyring <path> — the GPG keyring used for verification. Without it, signature verification does not happen; there is no default keyring and no implicit trust store.
--signature <source> — an additional signature source, for a signature you obtained out of band rather than from the server. Repeatable. Only usable with a positional collection name, not with -r.
--required-valid-signature-count <n> — how many signatures must verify. A positive integer, or all. Prefix with + to fail when no valid signatures are found at all — +all rather than all.
--ignore-signature-status-code <code> — tolerate specific GPG status codes such as NO_PUBKEY or EXPKEYSIG. Each one you add is a documented hole in the check.
--disable-gpg-verify — turn it off entirely.

What the checksum chain covers, and the one thing it does not

The chain is:

MANIFEST.json
  ├── collection_info   (namespace, version, licence, dependencies)
  └── file_manifest_file
        ├── name: FILES.json
        └── chksum_sha256: <hash of FILES.json>

FILES.json
  └── one entry per file: { name, ftype, chksum_type, chksum_sha256 }

Verifying walks it in that order: hash FILES.json and compare against MANIFEST.json, then hash every file and compare against FILES.json.

Two properties worth knowing:

Extra files are not covered. FILES.json enumerates what the build included. A file added to the installed directory afterwards has no entry to fail against. The check tells you the collection’s own files are unmodified — it is not a general-purpose directory integrity monitor, and a dropped-in extra module in the collection’s plugins/ tree is exactly the sort of thing it will not report.

The manifest is the trust anchor, and offline it is unauthenticated. --offline trusts the MANIFEST.json sitting next to the files. Only a signature, or a comparison against the server, gives that manifest any standing of its own.

Combine the two and you get the honest posture: offline verification is a change-detection tool for your own estate, GPG signatures are the authenticity control, and the two are complementary rather than alternatives.

What happens if verify fails on a production controller?

Treat it as an integrity incident until proven otherwise, and resist the urge to fix it by reinstalling — reinstalling destroys the evidence.

Do not run automation from that controller. Its plugins execute locally with your credentials; if the modification is hostile, so is anything you run next.
Record what changed. The verify output names the files. Copy them somewhere else before touching anything, along with stat output for timestamps and ownership.
Find out who. Shell history, sudo logs, file mtimes, and the change record. Nine times in ten this ends at a colleague’s 2am local patch, and the honest outcome is a process fix rather than an incident.
Reinstall from a pinned requirements.yml, then verify again and confirm clean.
If it was not a colleague, the controller is compromised, and Part XLVII’s controller-security material applies: the credentials on that machine reach the whole fleet, and rotating them is the first step rather than the last.

The reason to check regularly rather than during incidents is that the answer to “when did this change” is only useful if the last known-good timestamp is recent.

The two checksum files

Verifying, and watching it catch something

--offline versus online

GPG signatures

Knowledge check

`--offline` versus online