Anti-pattern: secrets in Git · Ansible

Ansible repositories are unusually attractive targets. A single clone can contain the credentials for an entire estate — database passwords, API tokens, the automation account’s access to every host — organised neatly by environment, with a README explaining what each one is for.

That is fine, and it is what vault is for. This lesson is about the four ways the protection gets defeated, and about why the remediation is much larger than people expect.

The four shapes

1. Plaintext, uncommented

Read-only / Safethe obvious form— Read-only: file contents, shown for recognition. Do not commit this.

# group_vars/production/main.yml
db_password: REPLACE_ME
api_token: REPLACE_ME
grafana_admin_password: REPLACE_ME

Easy to find, easy to lint for, and it is not the one that gets you.

2. The vault password committed alongside the vault

The variables are properly encrypted. ansible.cfg says vault_password_file = .vault_pass. And .vault_pass is in the repository, because somebody wanted CI to work.

The encryption is now decoration. Anyone with a clone has both halves.

3. A secret in a file nobody classified as a secret

The one that actually happens. It is not in group_vars; it is in:

A dynamic-inventory config file with an API token in it
A Makefile or wrapper script with export ANSIBLE_VAULT_PASSWORD=…
A test fixture with “fake” credentials that turned out to be real
A .env file that a .gitignore rule stopped covering when it moved
A commented-out line kept “just in case”
A CI configuration file with an inline token

4. A secret that was removed

Committed on Tuesday, removed on Wednesday, and still in the history — which is the property that makes this anti-pattern different from every other one in this part.

The failure: remediation is a fleet-wide rotation

This is the part that gets underestimated, because “delete the file and force-push” feels proportionate to “committed a file by mistake”.

The actual work:

Enumerate what leaked. Not just the obvious value - every credential in every commit in the affected range, because the same push often carried more than one.
Determine the blast radius per credential: which hosts, which services, which third parties accept it. A database password may be in six application configs; an API token may be in use by two other teams.
Rotate at the source: generate new credentials in the database, the cloud console, the vendor portal.
Distribute the new values to every consumer - which is an Ansible run, against production, under time pressure.
Retire the old credential, which is a separate step and the one most often skipped. A rotated-but-not-revoked credential is still valid.
Verify. Prove the old value no longer authenticates, per service, rather than assuming the revocation applied.
Rewrite the history if you wish, as hygiene. It is the last step and the least important one.

Step 4 runs under exactly the conditions Part XLIV describes

You are pushing new credentials to a fleet, at speed, because the old ones are public. Some hosts are unreachable. Some are mid-deployment. Some have a service that reads the credential at start-up and will not pick up the change without a restart you did not plan.

A partial rotation is the dangerous state: some hosts hold the new credential and some the old, and until you retire the old one, both work — including for whoever has the leaked copy. Retire it too early and you break the hosts that did not get the update.

This is the sequencing problem from Part XLIV, arriving unrehearsed, on a day chosen by an accident. Which is the actual argument for not committing secrets: not the leak itself, but that the fix is the hardest kind of change this course covers.

Prevention that works

A pre-commit hook that scans staged content. It catches the mistake before the push, which is the only point at which the cost is still small. Part XXXVIII covers the CI half; the local hook is what saves you.
Secret scanning in CI as the second line, so a bypassed hook is still caught - though by then it is pushed, and the clock has started.
A .gitignore for the vault password file plus a check that it is genuinely absent. .gitignore does not untrack a file that is already tracked, which is a common and expensive surprise.
The vault password from an environment variable or a file outside the repository, per Part XXI - and in CI, from the CI system secret store rather than from a file in the checkout.
A review habit: any diff that adds a long high-entropy string gets read carefully, whatever the filename.

Read-only / Safeaudit what is currently tracked— READ-ONLY: reads the index and history. Run it now rather than after an incident.

cd /srv/ansible

# Is the vault password file tracked, despite .gitignore?
git ls-files | grep -E 'vault[-_]?pass|\.vault'

# Files that should be encrypted but are not.
grep -rLE '^\$ANSIBLE_VAULT' --include='vault*.yml'    group_vars/ host_vars/ 2>/dev/null

# Any occurrence of a likely secret assignment across all history.
git log -p --all -S'password' -- '*.yml' | head -40

The same reflex, elsewhere: `host_key_checking = False`

This belongs in the same lesson because it comes from the same decision, made in the same five seconds: a control was inconvenient, so it was removed rather than configured.

Read-only / Safethe line, and what it costs— Read-only: configuration shown for recognition. This setting applies to every host in every run made from this controller.

[defaults]
host_key_checking = False

It appears because the first run against new hosts prompted for host key acceptance and the run was in CI where nothing could answer. That is a real problem with a real solution, and this is not it.

The failure. SSH host key verification is what detects a man-in-the-middle. Disabled, the controller will connect to whatever answers on that address and hand it the automation account’s credentials, the vault-decrypted variables for that host, and whatever become password the play uses. A DNS change, an ARP spoof, or an IP reassigned to a different machine is now indistinguishable from the real host.

Note the asymmetry with the secrets case: this one has no incident until it has a very large one, and no signal in between.

The corrected forms, in order of preference: populate known_hosts from a trusted source at provisioning time; use ssh-keyscan as an explicit, reviewed step whose output is checked in and versioned; or manage entries with ansible.builtin.known_hosts driven from inventory. Part XLVII covers all three, and Part XX covers which settings should never be weakened.

Under the hood

Why is a Git leak irrevocable in a way that, say, a leaked file on a fileserver is not?

Because of content addressing. Git objects are stored under the SHA of their content, and references — branches, tags — are pointers into that object store. Deleting a file in a new commit adds an object; it removes nothing. The blob containing the credential is still there, still addressable by its hash, and still reachable through the history that references it.

Even a history rewrite only makes the object unreferenced. It persists in the repository until garbage collection removes it, and remains directly fetchable by SHA in the meantime — many hosting platforms serve unreferenced objects by hash indefinitely, and several keep them in pull-request records that no rewrite touches.

Then there is distribution. Git is designed so that a clone is a complete copy. Every clone made between the push and the rewrite holds the object, and those clones are on laptops, in CI caches, in container build layers, and in whatever backup runs against them — none of which you can reach or enumerate.

The property that makes Git excellent for disaster recovery, which Part L relies on entirely, is exactly the property that makes a leak permanent. Full replication with immutable content-addressed history is the same mechanism in both directions. It is not a flaw to be worked around; it is the design, and the correct response is to keep secrets out rather than to plan on getting them back.

What happens if the leaked credential cannot be rotated quickly?

It happens: a vendor credential with a change-control process, a certificate with a manual issuance path, a shared account other teams depend on, or a database password embedded in an application that requires a release to change.

The reasoning has to shift from remediation to containment, and the questions are specific.

Can it be constrained rather than replaced? Frequently yes, and faster. An API token restricted to a source IP range, a database account restricted to specific hosts, a cloud key with its permissions reduced to what it currently needs. This shrinks the blast radius within minutes while the rotation goes through its process.

Can you detect its use? Turn on authentication logging for that credential and alert on any source you do not recognise. You cannot prevent the use, but you can stop being unaware of it — which changes this from an unknown to a monitored risk.

Who else must know? If the credential belongs to a third party or is shared with another team, they are affected and it is not your decision alone whether to accept the delay. That conversation goes badly later if it did not happen now.

What is the actual exposure window? Between push and rotation, and it should be written down. A leak that sat in a public repository for eight months is a different incident from one caught in twenty minutes, and the response should differ.

Record the deadline and own it. “Cannot be rotated quickly” reliably becomes “was never rotated” without a date and a name attached. This is the same discipline as the break-glass rotation step in Part L lesson 4, and it fails the same way when it is left implicit.

Knowledge check

Why this matters

Every other anti-pattern in this part degrades something. This one is the only one that cannot be undone.

A shell task can be converted. A missing --limit can be added. An inaccurate changed can be corrected and the fleet re-verified. A committed credential is public from the moment of the push, and no action available to you afterwards changes that — which is why the response is a rotation and not a repository edit.

The remediation is also the hardest change type in this course. You are pushing new credentials to a production fleet, at speed, with partial reachability and services that will not pick up the change without a restart, and there is a window during which both the old and new values work, including for whoever has the leaked copy. Nobody rehearses this, and it arrives on a day chosen by an accident.

Which puts almost all the value in prevention: a pre-commit hook, a vault password that lives outside the checkout, and a review habit of reading any diff that adds a long high-entropy string. Ten minutes of setup against an incident that consumes a week.