Lab: Leak hunt — find every place the credential escaped · Lab · Ansible

Objective

By the end of this lab you will have followed one fake credential through a single playbook run and found every place a copy of it came to rest. There are more than you expect, several survive no_log, and two of them are files a colleague can read without any Ansible privileges at all.

Architecture

One play, one credential, one canary string. The whole point is that nothing exotic is involved: this is what an ordinary deployment task does.

              vault (encrypted at rest)
                     │  decrypted into controller memory
                     ▼
   ┌─────────────────────────────────────────────┐
   │ 1. debug / task output                       │
   │ 2. module arguments at -vvv                  │
   │ 3. module return value (stdout)              │
   │ 4. --diff output                             │
   │ 5. ANSIBLE_LOG_PATH log file                 │
   │ 6. the backup: true file on the managed node │
   │ 7. shell history / process arguments         │
   └─────────────────────────────────────────────┘

Requirements

A controller with ansible-core 2.21.x. Output below captured from 2.21.3.
No managed nodes required for the controller-side leaks; the play runs against ansible_connection: local. To reproduce leak 6 — the backup file on the target — a managed node is more honest, so B-nested is listed as the higher-fidelity mode.
No privilege escalation.

Scenario

An auditor asks a simple question: “when the deployment playbook runs, how many copies of the database password exist, and where?”

The team’s answer is “one, in the vault”. Your job is to find out how many there really are.

Tasks

Task 1: Set up the canary

WORKDIR="$HOME/ansible-leakhunt-lab"
mkdir -p "$WORKDIR"
cd "$WORKDIR"

CANARY='CANARY-REPLACE-ME-0f3d9c'
echo "$CANARY" > canary.txt

inventory.yml:

all:
  hosts:
    node1:
  vars:
    ansible_connection: local

deploy.yml — an ordinary deployment play, written the way most are:

- name: Deploy the application configuration
  hosts: all
  gather_facts: false
  vars:
    db_password: CANARY-REPLACE-ME-0f3d9c
    app_dir: "{{ playbook_dir }}/target"

  tasks:
    - name: Ensure the target directory exists
      ansible.builtin.file:
        path: "{{ app_dir }}"
        state: directory
        mode: '0755'

    - name: Report what we are about to deploy
      ansible.builtin.debug:
        msg: "configuring database access as {{ db_password }}"

    - name: Write the application configuration
      ansible.builtin.copy:
        content: |
          [database]
          host = db.example.com
          password = {{ db_password }}
        dest: "{{ app_dir }}/app.conf"
        mode: '0600'
        backup: true

    - name: Verify the credential landed
      ansible.builtin.command: "/bin/grep -c password {{ app_dir }}/app.conf"
      changed_when: false

    - name: Register with the application
      ansible.builtin.command: "/bin/echo registering with {{ db_password }}"
      changed_when: false

Task 2: Run it the way a person runs it, and hunt

cd "$HOME/ansible-leakhunt-lab"
CANARY='CANARY-REPLACE-ME-0f3d9c'

# The run a person actually types when something is not working
ANSIBLE_LOG_PATH="$PWD/ansible.log" \
  ansible-playbook -i inventory.yml deploy.yml -vvv --diff > run-vvv.txt 2>&1

# Run it again so the backup file is created
ANSIBLE_LOG_PATH="$PWD/ansible.log" \
  ansible-playbook -i inventory.yml deploy.yml \
    --extra-vars "db_password=$CANARY-v2" --diff >> run-vvv.txt 2>&1

Now hunt. Do this systematically — the point of the lab is the method, not the answer:

CANARY='CANARY-REPLACE-ME-0f3d9c'

echo "=== every file under the working directory ==="
grep -rl "$CANARY" . 2>/dev/null

echo "=== count per file ==="
grep -rc "$CANARY" . 2>/dev/null | grep -v ':0$'

Read-only / Safecontroller— Every artefact holding a copy of the credential

$ grep -rc 'CANARY-REPLACE-ME-0f3d9c' . | grep -v ':0$'

./ansible.log:6
./canary.txt:1
./deploy.yml:1
./run-vvv.txt:9
./target/app.conf:1
./target/app.conf.20260811-2146.4471.2026-08-11@21:46:03~:1

Illustrative output

Task 3: Name each location and how it got there

Work through them. Do not skip to the fix — the classification is what you are learning.

1. The debug task output. The play printed it deliberately. One line, in every log of every run, forever.

grep -n 'configuring database access' run-vvv.txt

2. Module arguments at -vvv. Every module receives its parameters as a serialised structure, and -vvv prints that structure:

grep -n -B3 "$CANARY" run-vvv.txt | grep -E '"_raw_params"|"content"|"cmd"' | head

The content: of the copy task and the cmd: of both command tasks all carry the credential in plain text.

3. The module return value. command returns stdout, and the last task echoed the credential:

grep -n '"stdout"' run-vvv.txt | head

4. --diff output. The copy task’s diff shows the file content before and after — including the line with the password:

grep -n -A6 'before:.*app.conf' run-vvv.txt | head -12

5. The log file. ANSIBLE_LOG_PATH writes everything the callback would have printed, to a file:

Read-only / Safecontroller— Note the mode as well as the content

$ ls -l ansible.log && grep -c 'CANARY-REPLACE-ME-0f3d9c' ansible.log

-rw-rw-r-- 1 operator operator 4108 Aug 11 21:46 ansible.log
6

Illustrative output

The log file is created with your umask, not 0600

-rw-rw-r--. World-readable. Group-writable.

Ansible creates the log file with the process’s default umask and makes no attempt to restrict it, because it has no way of knowing whether the content is sensitive. On a shared controller — which is what a controller usually is — every credential that ever appeared in a run output is readable by every user on the box, in one file, indefinitely.

If you set log_path, you own the file’s permissions and its rotation. Create it 0600 before the first run, and put it somewhere with a retention policy:

LOGDIR="$HOME/.ansible-logs"
install -d -m 0700 "$LOGDIR"
install -m 0600 /dev/null "$LOGDIR/ansible.log"

6. The backup file on the target. backup: true wrote the previous version of the config beside the new one, on the managed node:

ls -l target/
cat target/app.conf.*~

The live file is mode 0600 because you asked for it. The backup is not covered by that mode: — it is written by the module’s backup step — so check it, and note that it persists forever unless something removes it.

7. Shell history and process arguments. The second run passed the credential on the command line:

history | grep -c "$CANARY" || true

And while a run is in flight, anything on the command line is visible in ps to every user on the machine:

$ ps -ef | grep ansible-playbook
operator  4471  ... ansible-playbook -i inventory.yml deploy.yml
          --extra-vars db_password=CANARY-REPLACE-ME-0f3d9c --diff

Record all seven in leak-inventory.md, each with the command that found it.

Task 4: Apply `no_log` and measure exactly what it fixes

    - name: Report what we are about to deploy
      ansible.builtin.debug:
        msg: "configuring database access"     # secret removed entirely

    - name: Write the application configuration
      ansible.builtin.copy:
        content: |
          [database]
          host = db.example.com
          password = {{ db_password }}
        dest: "{{ app_dir }}/app.conf"
        mode: '0600'
        backup: true
      no_log: true

    - name: Register with the application
      ansible.builtin.command: "/bin/echo registering"
      changed_when: false
      no_log: true

Re-run the hunt against a clean directory and compare.

Read-only / Safecontroller— What a no_log task looks like at -vvv

$ ansible-playbook -i inventory.yml deploy.yml -vvv | grep -A3 'TASK \[Write the application configuration\]'

TASK [Write the application configuration] *************************************
task path: /home/operator/ansible-leakhunt-lab/deploy.yml:20
ok: [node1] => {
  "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result"
}

Now measure what survives:

CANARY='CANARY-REPLACE-ME-0f3d9c'
grep -rc "$CANARY" . | grep -v ':0$'

Three of the seven survive no_log, and they are the ones that matter:

The file on the target. Obviously — it is the point of the task. Its mode is your responsibility.
The backup file. Same content, and no_log on the task does not change what the backup step writes to disk.
The command line and shell history, if you used --extra-vars.

What could go wrong?

What happens if the task fails while no_log: true is set?

You get a censored failure. The error message, the return code and the stderr are all suppressed, so you know a task failed and nothing else. This is the single biggest operational cost of no_log and the reason people turn it off “temporarily” and forget.

The mitigation is to keep the secret-bearing part of the work in the smallest possible task and leave the surrounding verification tasks loggable. A copy with no_log followed by a stat-and-assert without one gives you both the suppression and a usable failure.

What happens if no_log is set on a block?

It applies to every task in the block, including ones that carry no secret. That is usually more suppression than you want. Set it per task.

What happens if a secret ends up in a fact?

no_log does not follow a value. A registered result stored with set_fact and cacheable: true is written to the fact cache — a JSON file on the controller under fact_caching_connection if jsonfile is in use — with no suppression at all. Never set_fact ... cacheable: true on anything derived from a credential.

What happens if the callback plugin is not the default?

no_log is enforced at the task-result level, before callbacks see the result, so a third-party callback also gets the censored version. But a callback that writes its own artefact — a JUnit XML file, a JSON run record — can still capture task names and arguments you templated into them. A task named Set password for {{ db_user }} is fine; one named Set password {{ db_password }} is not.

Task 5: Harden what `no_log` cannot reach

The backup file. Either turn it off for secret-bearing files, or clean it up explicitly:

    - name: Write the application configuration
      ansible.builtin.copy:
        content: "{{ lookup('template', 'app.conf.j2') }}"
        dest: "{{ app_dir }}/app.conf"
        owner: appuser
        group: appuser
        mode: '0600'
        backup: false          # a backup of a credential file is a second credential file
      no_log: true

If you need the rollback capability that backup: true gives you, the answer is version control of the template, not a copy of the rendered secret sitting on the host forever.

The log file. Create it with a safe mode before Ansible does:

LOGDIR="$HOME/.ansible-logs"
install -d -m 0700 "$LOGDIR"
install -m 0600 /dev/null "$LOGDIR/ansible.log"

ANSIBLE_LOG_PATH="$LOGDIR/ansible.log" ansible-playbook -i inventory.yml deploy.yml
ls -l "$LOGDIR/ansible.log"

The command line. Replace --extra-vars with a vault:

umask 077
printf 'lab-vault-password-REPLACE_ME\n' > .vault-pass
chmod 0600 .vault-pass

ansible-vault encrypt_string --vault-password-file .vault-pass \
  --stdin-name 'vault_db_password' <<'EOF'
CANARY-REPLACE-ME-0f3d9c
EOF

Paste the result into group_vars/all/secrets.yml and consume it as db_password: "{{ vault_db_password }}".

Task 6: Build the scanner

The durable artefact from this lab is not the hardened play — it is the check that tells you when a future change reintroduces a leak.

cat > scan-for-secrets.sh <<'SCRIPT'
#!/usr/bin/env bash
# Fail if a known canary appears in any artefact this run produced.
# Usage: scan-for-secrets.sh CANARY_STRING DIR [DIR...]
set -euo pipefail

canary="${1:?usage: scan-for-secrets.sh CANARY DIR [DIR...]}"
shift
dirs=("$@")
[ "${#dirs[@]}" -eq 0 ] && dirs=(.)

found=0
while IFS= read -r hit; do
  echo "LEAK: $hit"
  found=1
done < <(grep -rl --binary-files=without-match "$canary" "${dirs[@]}" 2>/dev/null || true)

if [ "$found" -eq 1 ]; then
  echo "FAIL: the canary appears in the files listed above."
  exit 1
fi

echo "PASS: no occurrence of the canary in ${dirs[*]}"
SCRIPT

chmod 0755 scan-for-secrets.sh

Run it against your working directory, excluding the two files that are supposed to contain the canary:

CANARY='CANARY-REPLACE-ME-0f3d9c'

mkdir -p artefacts
mv run-vvv.txt ansible.log artefacts/ 2>/dev/null || true

./scan-for-secrets.sh "$CANARY" artefacts target

In CI, the equivalent runs against the build workspace after every job and fails the build. In a repository, the same idea belongs in a pre-commit hook — a secret scanner that runs before the commit rather than after the push.

Validation

The unhardened run produces at least six files containing the canary, and you can name how each one got it.
ls -l ansible.log shows a group- and world-readable mode.
target/app.conf.*~ exists after the second run and contains the canary.
With no_log: true on the copy task, a -vvv --diff run contains zero occurrences of the canary in its stdout.
With no_log: true, a deliberately failed version of that task reports a censored failure with no usable error message — confirm this, because it is the cost you are accepting.
After hardening, ./scan-for-secrets.sh "$CANARY" artefacts target reports leaks only for target/app.conf, which is the intended destination.
leak-inventory.md lists seven locations with a finding command each.

Expected Outcome

ansible-leakhunt-lab/
├── artefacts/{ansible.log, run-vvv.txt}
├── canary.txt
├── deploy.yml           <- hardened
├── group_vars/all/secrets.yml
├── inventory.yml
├── leak-inventory.md
├── scan-for-secrets.sh
├── target/app.conf      (mode 0600, no backup file)
└── .vault-pass          (mode 0600)

You can enumerate seven leak surfaces from memory, you know which three no_log does not close, and you have a scanner that turns “we think it is clean” into a check.

Troubleshooting

grep -r reports “binary file matches”. A .retry file or a fact cache can be non-UTF-8. grep -ra searches them as text; the scanner above uses --binary-files=without-match deliberately, so adjust it if you need binary coverage.

no_log: true and the task still prints the secret. The secret is in the task name, which no_log does not suppress. Task names are rendered by the callback before the result exists.

no_log did not suppress the diff. Check it is on the task and not on the play. no_log at play level applies to tasks, but a no_log placed under vars: or at the wrong indentation is silently ignored as an unknown key in some positions.

The canary appears in .ansible/tmp on the target. Module arguments are written to a temporary file on the managed node when pipelining is disabled. They are removed at task end, but a run interrupted between write and cleanup leaves them. pipelining = True in ansible.cfg avoids the temporary file entirely and is worth enabling for this reason as well as for speed.

The scanner passes but a secret is in git. This scanner checks the working directory as it is now. A secret committed and then removed is still in the object history. git log -p -S "$CANARY" finds it, and removing it means rewriting history — which is why the response to a committed secret is always to rotate first and clean up second.

Cleanup

This lab wrote a fake credential to several files. Every one of them must go, and the fact that the credential is fake is not a reason to be sloppy — the habit is what you are practising.

Step 1. Find every copy before deleting anything:

cd "$HOME/ansible-leakhunt-lab"
CANARY='CANARY-REPLACE-ME-0f3d9c'

grep -rl "$CANARY" . 2>/dev/null
grep -rl "$CANARY" "$HOME" --exclude-dir=ansible-leakhunt-lab 2>/dev/null \
  || echo 'no copies outside the lab directory'

The second command matters. If a copy escaped — into ~/.ansible.log, a .retry file, or a shell history file — you need to know before you delete the directory that would have reminded you.

This overwrites files in place and then unlinks them

The next block runs dd ... conv=notrunc over every file the grep found, destroying its content irrecoverably, and then shred -u on the vault password. There is no undo.

The grep -rl that feeds the loop is unanchored — it runs from the current directory, so cd "$HOME/ansible-leakhunt-lab" on the first line is load-bearing. Run it from anywhere else and it overwrites every file under that directory containing the canary. Confirm where you are and what matched before you run it:

cd "$HOME/ansible-leakhunt-lab" && pwd
grep -rl 'CANARY-REPLACE-ME-0f3d9c' . 2>/dev/null

Step 2. Overwrite the credential-bearing files, then remove them:

cd "$HOME/ansible-leakhunt-lab"

for f in $(grep -rl 'CANARY-REPLACE-ME-0f3d9c' . 2>/dev/null); do
  dd if=/dev/urandom of="$f" bs=4096 count=1 conv=notrunc status=none 2>/dev/null || true
done

shred -u .vault-pass 2>/dev/null || rm -f .vault-pass

Step 3. Clear the shell history entries the lab created. --extra-vars put the canary there in Task 2:

history | grep -n 'CANARY-REPLACE-ME' || echo 'nothing in the current history'

# Remove them from the in-memory history, then from the file
history -d "$(history | grep 'CANARY-REPLACE-ME' | head -1 | awk '{print $1}')" 2>/dev/null || true

grep -v 'CANARY-REPLACE-ME' "$HISTFILE" > "$HISTFILE.clean" 2>/dev/null \
  && mv "$HISTFILE.clean" "$HISTFILE" \
  && echo 'history file cleaned'

Step 4. Remove the working directory and verify:

rm -rf deletes the whole lab directory, including anything you added

The next block removes ~/ansible-leakhunt-lab and everything under it, recursively and without confirmation. If you saved notes, an inventory you want to keep, or a playbook you extended, copy it out first — the step above does that for the lab’s own deliverables and knows nothing about yours.

The path is written in full and anchored to $HOME deliberately. A relative rm -rf run from the wrong directory is how people delete work they meant to keep, and the shell gives no second chance.

Confirm what you are about to delete before you delete it:

ls -la "$HOME/ansible-leakhunt-lab"

mkdir -p "$HOME/ansible-lab-deliverables"
cp -a "$HOME/ansible-leakhunt-lab/scan-for-secrets.sh" \
      "$HOME/ansible-leakhunt-lab/leak-inventory.md" \
      "$HOME/ansible-lab-deliverables/"

rm -rf "$HOME/ansible-leakhunt-lab"

grep -r 'CANARY-REPLACE-ME-0f3d9c' "$HOME" 2>/dev/null \
  || echo 'no remaining copies under $HOME'

The deliverables you keep must not themselves contain the canary — check before copying, and edit leak-inventory.md to describe locations rather than to quote the string.

Under the hood

no_log is a task-level attribute enforced at two points, which is why its coverage is broader than most people assume and narrower than they hope.

On the managed node, AnsibleModule uses the no_log flags in the argument spec to sanitise its own return values before serialising them. This is how a module can return a result containing a password and have it arrive at the controller already redacted — and it only works for parameters the module declared no_log: true on.

On the controller, the task-level no_log: true replaces the entire result dictionary with the single censored key before any callback plugin sees it. That is why the diff, the module arguments and the return value all vanish together: they are fields of one object, and the object is replaced wholesale rather than filtered.

The consequences follow directly from that design. The task name is not part of the result, so it is not suppressed. The log file receives the callback’s output, so it receives the censored version — but everything that ran before you added no_log is already in it. And the file the module wrote on the target is not a result at all; it is the side effect the task existed to produce, and no amount of output suppression affects it.

What You Learned

One credential, seven resting places, from a play that does nothing unusual. Enumerating them is a five-minute grep and almost nobody does it.
no_log: true suppresses the whole task result — arguments, return value and diff together — because it replaces the result object rather than filtering it.
It does not suppress the task name, the file on disk, the backup file, or the command line. Three of the seven survive.
ANSIBLE_LOG_PATH creates a world-readable file with your umask, and it accumulates. If you set it, you own its mode and its retention.
backup: true on a secret-bearing file creates a second copy of the secret that nothing removes. Version-control the template instead.
--extra-vars is unfixable. Shell history, process table, CI job definition and CI logs, all at once.
The scanner is the durable artefact. A canary and a grep turn “we think it is clean” into something a pipeline can fail on.

Lab: Leak hunt — find every place the credential escaped

Objectives

Prerequisites

Objective

Architecture

Requirements

Scenario

Tasks

Task 1: Set up the canary

Task 2: Run it the way a person runs it, and hunt

Task 3: Name each location and how it got there

Task 4: Apply `no_log` and measure exactly what it fixes

Task 5: Harden what `no_log` cannot reach

Task 6: Build the scanner

Validation

Expected Outcome

Troubleshooting

Cleanup

What You Learned

Deliverables

Verification status

Objectives

Prerequisites

Objective

Architecture

Requirements

Scenario

Tasks

Task 1: Set up the canary

Task 2: Run it the way a person runs it, and hunt

Task 3: Name each location and how it got there

Task 4: Apply no_log and measure exactly what it fixes

Task 5: Harden what no_log cannot reach

Task 6: Build the scanner

Validation

Expected Outcome

Troubleshooting

Cleanup

What You Learned

Deliverables

Verification status

Task 4: Apply `no_log` and measure exactly what it fixes

Task 5: Harden what `no_log` cannot reach