Keeping secrets out of the repository

Part XXI covers how secrets leak out of a running playbook — through verbose output, through diff mode, through registered variables, through a log the automation account can write and a hundred people can read. This lesson is about the leak that happens before any of that: the credential that gets committed.

It deserves separate treatment because its remediation is different from every other defect in this course. For a bad task you fix the task. For a committed secret, fixing the file is not the fix, and the instinct to reach for git rm first is the thing that turns a contained incident into a slow one.

Two boundaries, two scanners

Secret scanning belongs in two places and they catch different things.

At commit time, on the developer’s machine, through a commit hook framework such as pre-commit. It scans the staged diff before the commit object exists. Catching it here means there is nothing to remediate: the secret was never in an object, never pushed, never mirrored.

At merge time, in CI, on the branch’s full diff against the base. This catches everything the commit hook did not, and the list of reasons it did not is longer than people expect: the hook was not installed, it was bypassed with --no-verify, the commit was made in a web editor, the commit was made by a bot, or the developer cloned before the hook was added.

The CI stage is the one that has to be right, because it is the one that cannot be skipped by the person committing. The commit hook is a convenience that saves an incident; the CI gate is the control.

What a generic scanner catches, and what it misses here

Generic secret scanners work on two signals: high-entropy strings, and patterns for known credential formats — cloud access keys, private key headers, tokens with a recognisable prefix. Any of the common tools in that category will do; the choice matters less than having one that runs on every branch.

What none of them reliably catches is the case specific to this repository: a file that should have been vault-encrypted and is not. A database password like Tr0ub4dor has no recognisable format and only moderate entropy. It looks like a word.

So write the gate that knows about your layout. The convention from the previous part — encrypted material lives in files named vault.yml — makes it three lines:

#!/usr/bin/env bash
# Fail if any file named vault.yml is not actually vault-encrypted.
set -uo pipefail
status=0
while IFS= read -r f; do
  if ! head -c 14 "$f" | grep -q '^\$ANSIBLE_VAULT'; then
    printf 'NOT ENCRYPTED: %s\n' "$f"
    status=1
  fi
done < <(find inventories -name 'vault.yml' -type f)
exit "$status"

$ bash scripts/check-vault-encrypted.sh

NOT ENCRYPTED: inventories/staging/group_vars/all/vault.yml

Exit status 1. The check relies on the vault header, which every encrypted file carries as its first line:

$ head -1 inventories/production/group_vars/all/vault.yml

$ANSIBLE_VAULT;1.2;AES256;prod

Two useful extensions, both cheap:

• Check the label too. A file under inventories/production/ whose header says dev is a real defect — it means production material was encrypted with the development password, so everyone with dev credentials can read it.
• Check that nothing outside the expected paths is encrypted. An encrypted file somewhere unexpected is usually someone working around a layout they did not understand, and it will not be decryptable by the job that needs it.

When one gets through

This is the section to read before you need it, because the ordering is counter-intuitive under pressure and every minute spent on the wrong step is a minute the credential is still valid.

1. Rotate the credential. Now, before anything else. From the moment it was pushed you must assume it is disclosed - the object is on the forge, in every clone, in every fork, in CI caches, in backups, and possibly in a search index. Rotation is the only step that changes what an attacker can do.
2. Confirm the new credential works and the automation is running with it. A rotation that breaks production at 03:00 is a second incident on top of the first.
3. Revoke or invalidate the old credential explicitly. Rotating to a new value does not always disable the old one - API tokens, SSH keys and service accounts frequently permit both until the old one is deleted.
4. Check for use. Search the authentication logs for the exposed credential over the whole window from first push to revocation. This is what tells you whether you had a leak or a breach, and it is the question your security team will ask first.
5. Only now consider the history. Decide whether to rewrite, knowing what that does and does not achieve - see below. For a public repository it is usually worth it; for an internal one it is often not worth the disruption.
6. Fix the gate that let it through. A scanner rule that did not match, a file not covered by the vault check, a path excluded from scanning. Without this step the same class of secret arrives again next quarter.
7. Write it down. What was exposed, for how long, who was notified, what was rotated, and what changed in the pipeline. This is the record that stops the third occurrence.

Knowledge check