Skip to main content
RunBook Academy

← All break/fix scenarios in Ansible

advancedsecrets~35 min

Break/Fix: a vaulted database password is sitting in a CI log that 400 people can read

Reported symptoms

  • ●A secret-scanning tool flags the CI build log for the deployment pipeline
  • ●The flagged string is the production database password, in full, in plain text
  • ●The vault file is intact, correctly encrypted, and has not been modified
  • ●No task in the playbook prints the password, and nobody added a debug task
  • ●Running the same playbook by hand from a workstation produces no such output
  • ●The pipeline has been producing this output on every run for eleven days
  • ●Build logs are readable by everyone with access to the CI system

Evidence

  • · The build log contains a JSON task result including the assembled connection string
  • · The same playbook run locally with default verbosity produces no such line
  • · The same playbook run locally with `-v` reproduces it exactly
  • · `git log -p .gitlab-ci.yml` shows `-v` added to the ansible-playbook invocation eleven days ago
  • · `git log -p roles/app/tasks/main.yml` shows `no_log: true` removed from a `set_fact` task in the same period
  • · The vault file header and modification time are unchanged
  • · Build log retention is 90 days and the logs are readable by every user of the CI system
Diagnosis and resolutionclick to reveal

Root cause

Two ordinary changes combined into a disclosure. A `set_fact` task assembles a database connection string from a vaulted password; it used to carry `no_log: true`, and that line was removed during a refactor by somebody who was debugging a templating problem and did not put it back. Separately, the pipeline definition gained a `-v` flag to make failures easier to read. Either change alone is invisible. Together they are a disclosure, because at verbosity 1 and above Ansible prints the full result of every task, and the result of a `set_fact` includes the fact it set. The password was never decrypted anywhere it should not have been and the vault is entirely intact - the leak is in the callback output, which is the part of Ansible that nobody thinks of as a place secrets can escape to. It stayed hidden because the local reproduction used default verbosity, so the playbook appeared clean on a workstation and only leaked inside CI.

Remediation

Treat the credential as disclosed and rotate it first; everything else is cleanup. Assume any log that existed is copied, indexed and cached, so deleting the build log reduces further exposure but does not undo the disclosure. Then close both halves: restore `no_log: true` on the task that handles the secret and on every task downstream that consumes a value derived from it, and remove the blanket `-v` from the pipeline in favour of raising verbosity only for a specific failing run. Purge the affected build logs and any artefact store or log-shipping destination that received a copy, and check whether the logs were forwarded anywhere with different retention.

Verification

Run the pipeline at the same verbosity it uses in production and grep the captured output for the credential; it must not appear. Then prove the check can fail by running once at `-vvv` against a scratch target with a deliberately planted marker value and confirming the grep finds it, so you know the search works before trusting a negative result. Confirm the old credential no longer authenticates rather than confirming a new one was issued. Confirm the `no_log` coverage by inspecting the task results in a verbose run - the censored placeholder should appear where the secret would have been.

Prevention

Do not raise verbosity globally in automation. Verbosity is a debugging tool for a specific run, and a pipeline that always runs verbose will eventually print something it should not. Put `no_log: true` on every task that handles a secret and on every task that handles a value derived from one, because the protection follows the task rather than the variable. Scan build logs for secrets automatically, treat a hit as an incident with rotation rather than a warning to be triaged, and keep retention short. Review the removal of `no_log` as carefully as the addition of a credential - it is a one-line diff that turns a protected task into an unprotected one and it reads as noise. Finally, prefer fetching credentials at the point of use over assembling them into facts that persist for the whole play.

Reported symptoms

A secret scanner reports a high-severity finding against the deployment pipeline’s build log. The flagged string is the production database password.

The security team’s first questions all get reassuring answers, which is what makes the next hour confusing:

  • Is the vault file committed in plain text? No. It is encrypted and its header and modification time are unchanged.
  • Did somebody decrypt it into the repository? No.
  • Is there a debug task printing the password? No. There is no debug task anywhere in the role.
  • Can it be reproduced? A developer runs the same playbook from a workstation against a staging target. The output is clean.

At that point the theory shifts to a compromised CI runner, and an hour goes into examining a machine that is behaving perfectly correctly.

Eleven days of build logs contain the credential. Roughly 400 people can read them.

Evidence provided

Read-only / Safethe full task result, including the fact it set
$ grep -n -m1 'app_db_dsn' build-4471.log
ok: [app01] => {"ansible_facts": {"app_db_dsn": "postgresql://app:REPLACE_ME_DB_PASSWORD@db-prod.example.com:5432/app"}, "changed": false}
Read-only / Safethe local reproduction that made everyone look at the runner
$ ansible-playbook -i inventory deploy.yml --limit staging01 | grep -c app_db_dsn
0
Read-only / Safethe same playbook, one flag different
$ ansible-playbook -i inventory deploy.yml --limit staging01 -v | grep -c app_db_dsn
1
Read-only / Safechange one: make failures easier to read
$ git log -p --since=14.days -- .gitlab-ci.yml | grep -E '^[-+].*ansible-playbook'
-    ansible-playbook -i inventory deploy.yml
+    ansible-playbook -i inventory deploy.yml -v
Read-only / Safechange two: removed while debugging a templating problem, never restored
$ git log -p --since=14.days -- roles/app/tasks/main.yml | grep -E '^[-+].*(no_log|set_fact|app_db_dsn)'
   ansible.builtin.set_fact:
   app_db_dsn: "postgresql://app:{{ vault_db_password }}@{{ db_host }}:5432/app"
-  no_log: true
Read-only / Safethe vault is intact and untouched - which is true and irrelevant
$ ansible-vault view --vault-password-file /etc/ansible/vault-pass group_vars/production/vault.yml | head -1; stat -c '%y' group_vars/production/vault.yml
vault_db_password: REPLACE_ME_DB_PASSWORD
2026-05-14 09:22:41.000 +0000

Work the evidence before reading on

The vault is fine. The runner is fine. Nothing prints the password on purpose.

  1. The local run is clean and the CI run is not. List every difference between the two invocations. There is exactly one that matters.
  2. set_fact sets a fact. What does the result of a task look like at verbosity 1, and what does the result of a set_fact task contain?
  3. Two commits landed in the same fortnight. Which one is the vulnerability and which one is the exploit?

Before continuing: if you restored no_log: true and left -v in place, would you be safe? If you removed -v and left no_log off, would you be safe? What does that tell you about how to think about either change on its own?

Root cause

1. Verbosity prints the full result of every task

At default verbosity Ansible prints a status word and a host name. From -v upwards it prints the complete result dictionary the module returned.

For most modules that is dull. For set_fact it is the fact itself:

ok: [app01] => {"ansible_facts": {"app_db_dsn": "postgresql://app:REPLACE_ME_DB_PASSWORD@..."}, "changed": false}

Nothing here is a bug. The callback is doing precisely what verbosity asks it to do, and the module result genuinely contains the value the module was asked to set.

2. no_log protects a task, not a variable

no_log: true suppresses the result of the task it is written on. It is not a taint marker that follows a value around.

The set_fact that assembles the connection string had it, and lost it in a refactor. From that moment the task’s result was printable, and the value it contained was the credential.

The reverse mistake is just as common. no_log on the task that reads a secret does not protect a later task that consumes it: a debug on a derived variable, a command with the value on its argument line, or a uri task that echoes a failed request body will all print it. Protection has to be applied at every task that touches the value.

3. Neither change was reviewable in isolation

The -v in the pipeline definition is a one-word diff in a YAML file, proposed to make failing builds easier to read, and it is a completely reasonable request.

The no_log removal is a one-line deletion inside a role, made while debugging a templating problem, in a commit that also changed the thing being debugged. Both passed review because both are individually sensible and neither reviewer could see the other.

The combination is a disclosure of a production credential to everyone with CI access, held for 90 days by the log retention policy.

Resolution

  1. Rotate the credential first. Everything after this is cleanup, and the cleanup does not undo the disclosure. Treat the value as known to everyone who has had CI access for eleven days.
  2. Establish the true exposure before deleting anything. How many builds, over what period, and where else did the logs go - artefact stores, log shipping, chat notifications, email on failure, screenshots in tickets.
  3. Restore no_log: true on the set_fact task, and audit every task downstream that consumes a value derived from the credential. Protection is per task, so a single restoration is unlikely to be sufficient.
  4. Remove -v from the pipeline definition. If verbose output is needed to diagnose a specific failure, raise verbosity for that run, deliberately, and account for where the output lands.
  5. Purge the affected build logs and every copy you identified. Do this after the rotation, not instead of it.
  6. Verify the old credential is refused by the database rather than verifying that a new one was issued. Rotation is not complete until the old value fails.
  7. Notify according to policy. A production credential readable by 400 people for eleven days is a reportable disclosure in most organisations regardless of whether anyone read it.
  8. Add automated secret scanning to build logs, and treat a hit as an incident with rotation rather than a warning to triage. This finding came from a scanner that already existed; the gap was in what happened next.

Verification

  1. The pipeline no longer emits the value. Run it at the verbosity it uses in production, capture the output, and grep for the credential; the count must be zero.
  2. The search itself works. Plant a distinctive marker value in a scratch run at -vvv, and confirm the same grep finds it. A negative result from an untested search is not evidence, and this is the check that can fail.
  3. The censoring is visible where it should be. In a verbose run against a scratch target, the protected tasks show the censored placeholder rather than a result, including in a loop and including on a deliberately failed task.
  4. The old credential is refused. Attempt authentication with it and require a failure. A new credential existing is not the same as the old one being dead.
  5. Every copy is accounted for. The exposure inventory is closed: build logs, artefact stores, log-shipping destinations, notification channels, and anything with a longer retention than the CI system itself.
  6. The scanner covers this path. Reintroduce a marker into a branch build and confirm the scanner fires and that the response is rotation rather than a triage note.
  7. A review rule exists. Removing no_log now requires explicit approval, and there is a test that fails if the protected tasks lose it.

Prevention

  • Never run automation at raised verbosity by default. Verbosity is for a specific investigation, with a known destination for the output.
  • Apply no_log to every task that handles a secret and every task that handles anything derived from one. It protects a task, not a value, and a derived value is just as sensitive as the original.
  • Review no_log removals as security changes. A one-line deletion inside a refactor is the most likely way this protection is lost, and it reads as noise in a diff.
  • Prefer fetching a credential at the point of use to assembling it into a fact. A fact persists for the play and is available to every subsequent task result, which is a wide surface for something needed by one task.
  • Scan build logs continuously, keep retention short, and treat any hit as a disclosure requiring rotation. Logs are the least-guarded place secrets go, and they are copied, indexed and cached by systems nobody lists.
  • Reproduce faithfully when investigating. The local run was clean because it differed from the pipeline in one flag, and that difference sent an hour of the investigation at an innocent machine.