RunBook AcademyAnsible

← All checklists in Ansible

QuarterlySecurity

Checklist: Ansible secrets review

18 items ·13 critical ·4 warn ·1 info

Run this quarterly, and additionally whenever somebody leaves the team, a credential is suspected of exposure, or a new external secret store is introduced. Run it from a clean checkout of the default branch, on a machine that holds the production vault password - the decryption item cannot run anywhere else.

Set VAULT_PASS_FILE to the path of your vault password file before running the decryption sweep; the command reads it from the environment rather than having a path baked in.

What a failure means

There are two shapes of failure here and they need different responses.

A finding in the working tree - an unencrypted vault file, a task with no no_log, a plaintext default - is a defect to fix. Fix it and move on.

A finding in the log, the shell history or the Git history is an exposure. The credential has already been written somewhere it should not be, and cleaning up the file does not undo that. Rotate the credential; treat the cleanup as secondary.

Access this needs

The repository scans need a checkout only. The decryption sweep needs the vault password. The log scan needs read access to the Ansible log, which on most controllers means the automation account. Nothing here writes anything, and the two commands that could print a secret - the log scan and the shell history scan - both mask the value before printing.

Where the evidence goes

Record the date, the person who ran it, the findings and their disposition in the security review record. For any item that is attested rather than commanded

the holder list, the escrow, the leaver removals - record who attested it. Those three are the ones an auditor will ask about, and “we check it” is not an answer without a name and a date.

Sign-off

Reviewer: ________________ Date: ___________
Security owner: ___________ Date: ___________

Critical13 items

No credential is assigned a literal value anywhere in the repository. Prints every line that assigns a secret-shaped key to something other than a Jinja reference or a lookup; a `{{ vault_db_password }}` is not a finding, a quoted string is.
```
grep -rInE '^[[:space:]]*[a-z0-9_]*(pass|passwd|password|secret|token|api[-_]?key|private[-_]?key|credential)[a-z0-9_]*[[:space:]]*:[[:space:]]*[^[:space:]{]' \
  --include='*.yml' --include='*.yaml' --include='*.j2' --exclude-dir=.git .
```
Every file named as a vault file actually begins with the vault header. A file called `vault.yml` that somebody edited without `ansible-vault edit` is a plaintext credential store with a reassuring name. Prints one line per file that is not encrypted; no output is the pass condition.
```
find . -path ./.git -prune -o -type f \( -name '*vault*' -o -path '*/vault/*' \) -print \
  | while read -r f; do
      head -c 14 "$f" | grep -q '\$ANSIBLE_VAULT' \
        || echo "FINDING: $f is named as a vault file but is not encrypted"
    done
```

Every encrypted file can be decrypted with the vault password the controller actually holds. This catches the vault-id mismatch that otherwise surfaces as a failed production run: the file is encrypted, the password is valid, and they are not the same pair.

grep -rlZ --include='*.yml' '^\$ANSIBLE_VAULT' inventories/ group_vars/ host_vars/ 2>/dev/null \
  | xargs -0 -r -n1 sh -c '
      ansible-vault view --vault-password-file "$VAULT_PASS_FILE" "$1" >/dev/null 2>&1 \
        || echo "FINDING: $1 cannot be decrypted with the configured vault password"' sh

Every task that handles a secret sets `no_log: true`. Without it the value appears in the task result at -vv, in the log file, and in any callback that ships results elsewhere. Prints one line per task that touches a secret-shaped value without no_log.

python3 - <<'PY'
import glob, re, yaml
SECRET = re.compile(r'(pass|passwd|password|secret|token|api[-_]?key|private[-_]?key|credential)', re.I)
KEYS = ('tasks', 'pre_tasks', 'post_tasks', 'handlers', 'block', 'rescue', 'always')
def tasks(node):
    if isinstance(node, list):
        for i in node: yield from tasks(i)
    elif isinstance(node, dict):
        for k in KEYS:
            yield from tasks(node.get(k) or [])
        if 'hosts' not in node:
            yield node
for f in sorted(glob.glob('playbooks/**/*.yml', recursive=True)
                + glob.glob('roles/**/tasks/*.yml', recursive=True)
                + glob.glob('roles/**/handlers/*.yml', recursive=True)):
    try: doc = yaml.safe_load(open(f))
    except Exception: continue
    for t in tasks(doc):
        if not isinstance(t, dict) or t.get('no_log'): continue
        if SECRET.search(yaml.safe_dump(t)):
            print(f"FINDING: {f}: task {t.get('name')!r} handles a secret without no_log")
PY

No secret is passed as an argument to `command`, `shell` or `raw`. Arguments are visible in the process list on the managed node for the life of the task, so any local user can read them, and `no_log` does not hide them there. Use a module parameter, an environment variable or a file instead.
```
grep -rInE '(command|shell|raw):.*(pass|password|token|secret|key)=' playbooks/ roles/
```
No production secret has been passed on the command line with `--extra-vars`. The value lands in the shell history of whoever ran it and in the process list of the controller. Prints any such invocation found in the shell history, with the value masked.
```
grep -hE 'ansible(-playbook)?\b.*(--extra-vars|[[:space:]]-e[[:space:]]).*(pass|secret|token|key)' ~/.bash_history 2>/dev/null | sed 's/=[^ ]*/=REDACTED/g'
```

The Ansible log holds no credentials. A single task that was missing `no_log` six months ago put a production password into a file that is now on a backup tape. Prints any matching line with the value masked; no output is the pass condition.

p=$(ansible-config dump | sed -n 's/^DEFAULT_LOG_PATH([^)]*) = //p')
[ -r "$p" ] && grep -nIE '(password|passwd|secret|token|api_key|private_key)"?[:=]' "$p" \
  | sed 's/\(password\|passwd\|secret\|token\|api_key\|private_key\)"\?[:=][^,}"]*/\1=REDACTED/gI'

The Git history has been scanned, not just the working tree. Deleting a secret in a commit does not remove it from the repository - every clone still has it, and rewriting history does not recall the clones. Any secret ever committed must be treated as compromised and rotated.
```
git log --all -p -S 'password' -- inventories/ group_vars/ host_vars/ | grep -nE '^\+.*(password|secret|token)' | head -50
```
The pipeline runs a secret scanner on every push, and its failure blocks the merge. A review that depends on a human noticing a base64 blob in a 400-line diff is not a control.
```
grep -rn 'gitleaks\|trufflehog\|detect-secrets\|secret' .gitlab-ci.yml .github/workflows/ 2>/dev/null
```
Production and non-production use different vault identities and different passwords. One shared vault password means every engineer who can decrypt a staging variable can decrypt the production one, and it means rotating production forces a rotation everywhere.
```
ansible-config dump | grep -E '^DEFAULT_VAULT_IDENTITY_LIST|^DEFAULT_VAULT_ID_MATCH'; grep -rn 'vault_id' ansible.cfg playbooks/ 2>/dev/null
```
The list of people and systems that hold each vault password is written down and current. If nobody can produce that list, the honest answer to "who can read production credentials" is "we do not know", and no rotation schedule fixes that.
The production vault password is recoverable if the person who set it is unavailable - held in a break-glass store, not only in one password manager or one head. Verify the escrow copy opens, do not assume it.
Everyone who has left the team since the last review has been removed from the holder list, and every password they held has been rotated. Removing their SSH access does not un-know a vault password they memorised or copied.

Warning4 items

DEFAULT_VAULT_ID_MATCH is True where multiple vault identities are in use. With it False, Ansible tries every password it holds against every encrypted file - so a file encrypted with the staging key decrypts happily on a controller that also has the production key, and the separation you configured is not enforced.
```
ansible-config dump | awk '/^DEFAULT_VAULT_ID_MATCH/ && !/= True/ {print "FINDING: " $0}'
```
Every credential has a recorded rotation date and an owner, and none is past its interval. A secret with no rotation date is a secret that has never been rotated.
A vault rekey has been performed at least once, on a copy, and timed. `ansible-vault rekey` re-encrypts every file, which is a large commit and a coordination problem across open branches - discovering that during an incident is the wrong time.
```
grep -rlZ --include='*.yml' '^\$ANSIBLE_VAULT' . 2>/dev/null | xargs -0 -r -n1 echo "would be rekeyed:"
```
Fact caching does not write credentials to disk. With a `jsonfile` or `redis` cache plugin, any fact a task registered - including one containing a secret - persists outside the run and outside `no_log`.
```
ansible-config dump | grep -E '^CACHE_PLUGIN|^FACT_CACHING'
```

Info1 item

Where secrets come from an external store rather than vault, the automation authenticates per run with a short-lived credential, and that credential is not itself sitting in the repository. Otherwise the external store has moved the problem, not solved it.
```
grep -rn 'lookup(.*hashi_vault\|community.hashi_vault\|aws_secret\|azure_keyvault' playbooks/ roles/ 2>/dev/null
```