The ways secrets actually leak

This part is about Ansible Vault, external secret managers and no_log. None of that appears in this lesson.

The order is deliberate. Vault is a good tool applied to one specific problem — data at rest in a repository — and almost every serious secrets incident in an automation estate happened somewhere else. Teams encrypt the vars file, feel finished, and then leak the same credential through a debug task, a run log or a CI artifact, none of which Vault ever touched.

So before any tooling: where does a secret actually go?

Every credential in this lesson is REPLACE_ME or REDACTED, and every host is in documentation space. The routes are real. The material is not.

The secret’s journey

A credential in an automation workflow travels a fixed path, and each hop is a place it can escape:

repository  ->  controller memory  ->  module arguments  ->  target
    |                  |                     |                 |
 committed         run log            temp file on         service log
 in history        --diff             the managed node      backup file
 in clones         debug output       process table         config it wrote
 in CI cache       verbose output
                   shell history
                   CI job output

Vault protects exactly one box: the repository, and only while the data is at rest in it. Every other arrow is a separate control or no control at all.

Work through them, because the ones people miss are consistently the same ones.

1. Plaintext in Git — including after you delete it

A credential committed in plaintext is not removed by a commit that deletes the file. Git keeps the object. git log -p will show it, git show on the old commit will print it, and — critically — every clone already has it.

That last point is the one that changes the response. Cleaning your copy does nothing about:

• every developer’s clone and every stale branch
• the CI system’s cached checkout
• forks and mirrors, including any backup of the hosting service
• the hosting provider’s own storage, which a push --force may not clear
• anyone who was watching, including automated scrapers that index public and semi-public repositories continuously

2. A debug of a variable, or of a registered result

The most common route, and it does not look like a mistake while you are writing it.

$ ansible-playbook -i localhost, leak.yml

TASK [Leaks it - a debug of the variable] **************************
ok: [localhost] => {
  "db_password": "REPLACE_ME"
}

TASK [Print the registered result] *********************************
ok: [localhost] => {
  "built.ansible_facts.connection_string":
    "postgresql://svc:REPLACE_ME@db1.example.com/app"
}

The first is obvious in review. The second is the one that ships.

Nobody writes debug: var=db_password into a production playbook on purpose. What they write is debug: var=result while working out why a task behaves oddly — and result is the return value of a task that handled the credential, so the credential is in it. A connection string, a templated config, an API response echoing back the token it was given, a command line in cmd.

The rule that survives contact with reality: treat the registered result of any task that touched a secret as containing that secret, and never debug it whole. Print the one field you need.

3. --diff, which exists to show you file contents

--diff prints what changed inside files a task modified. That is exactly what it is for, and it is the reason it is dangerous around templates.

$ ansible-playbook -i inventory site.yml --check --diff

--- before: /etc/app/database.conf
+++ after: /etc/app/database.conf
@@ -1,3 +1,3 @@
[database]
host = db1.example.com
-password = OLD_REDACTED
+password = REPLACE_ME

The trap is that --check --diff is the responsible thing to do. It is the pre-flight everyone is told to run, it is what a careful operator does before a change, and it is frequently what CI runs on every pull request — where the output is retained as build logs that a much wider group can read than could read the repository.

--check --diff on a play that templates a credential prints that credential. Check mode makes the run harmless to the fleet. It does nothing about the output.

4. log_path, which captures everything the run printed

Set log_path and Ansible writes the run to a file on the controller. Verified on 2.21.3, with the leaky play from above:

$ ANSIBLE_LOG_PATH=$PWD/run.log ansible-playbook -i localhost, leak.yml >/dev/null; grep -c REPLACE_ME run.log; ls -l run.log

3
-rw-rw-r-- 1 ebrandi ebrandi 4096 Aug 11 21:34 run.log

Two things there deserve to be uncomfortable.

Three occurrences, from a play with two obviously leaky tasks. One of them came from a task that had no_log: true set — the next lessons deal with that specifically, and it is not what most people expect.

Mode 0664. The log is created with the process umask and Ansible does not restrict it. This controller has umask 0002, so the file is world-readable: every account on the machine can read every secret every run printed. Nothing warned about it, and nothing in the Ansible configuration mentions permissions.

If you enable log_path — and it is genuinely useful for audit — put it in a directory that only the automation account can enter, and set the umask for the account that runs it. A restrictive directory is the stronger of the two controls, because it survives somebody forgetting the umask.

5. -vvv, and the debugging session that gets pasted

Verbose output prints module arguments. The arguments are where the credential is. This is the exact moment when the pressure to share is highest — the run is failing, somebody asks for the output, and the fastest path is to paste it.

Which produces route 9, below, and is why the secret-handling rule has to be a habit rather than a policy: by the time you need it, you are debugging something else.

6. The controller’s shell history

ansible-playbook site.yml -e "api_token=REPLACE_ME"

That line is now in ~/.bash_history, and it was in the process table while it ran, where any user on the machine could read it with ps.

The become part of this course covers -e for passwords in detail and ranks it as the worst of the available routes. The point that belongs here: history is durable, is backed up, is rarely reviewed, and is preserved by a shell exiting normally — so the leak outlives the session and the shell, and nobody ever looks at it again.

7. Files on the managed node

Three routes, all on the target rather than the controller.

The module wrapper. Without pipelining, the module and its arguments are written to a temporary file on the managed node before execution. An interrupted run leaves it. ANSIBLE_KEEP_REMOTE_FILES leaves it on purpose, which is exactly what somebody sets while debugging the task that handles the credential. Pipelining removes this one entirely, as the SSH part covers.

backup: true. copy, template and lineinfile will keep a copy of the previous file next to the new one. That is a good operational habit and it means old credentials accumulate on the target, under a timestamped name, usually with permissions copied from a file that was expected to be replaced rather than retained. A rotation that leaves five generations of backup config on every host has not really rotated anything.

The service’s own logs. Once you hand a credential to a program, its handling is that program’s business. Plenty of software logs its configuration at startup, or logs a connection string on failure. Nothing in Ansible reaches into that.

8. Everything CI touches

CI concentrates all of the above and adds distribution.

• Job output is retained, searchable, and readable by a larger group than the repository — often by everybody in the organisation.
• Artifacts persist. A saved run.log, a JSON callback output, a diff report.
• Environment variables appear in the job configuration, which is frequently in a file in the repository.
• Caches keep checked-out working trees, including branches nobody remembers.
• Forks and pull requests from outside the trusted set may be able to run pipelines with access to secrets, depending on configuration.

9. The paste

A ticket, a chat channel, a shared document, a screenshot.

This is not a technical route and it is one of the most common. Ticket systems are searchable, retained for years, backed up, and readable by support staff, contractors and anyone who inherits the project. A credential pasted into a ticket in 2021 is still there.

Knowledge check