Where the vault password lives · Ansible

Every lesson so far has moved a secret behind a password. This one asks the obvious follow-up question, which is remarkably often the one nobody asks:

Where is that password?

Encryption does not remove a secret. It replaces many secrets with one, and the security of the whole arrangement collapses to the security of the replacement. If the vault password is easier to obtain than the credentials it protects, the encryption has made things worse — it has added a step and a false sense of completion.

The arrangement that protects nothing

This is what a great many estates actually have:

/srv/automation/
    ansible.cfg
    inventory/
        production/group_vars/all/vault.yml     0644, encrypted
    .vault-pass                                 0644

Or the same thing with vault_password_file = .vault-pass in ansible.cfg, and .vault-pass in .gitignore so it is at least not committed.

Anyone who can read the repository directory can read the password. Which means anyone who can read the repository can decrypt the vault. Which means the encryption protects the repository against exactly one thing: being copied without its directory — a git clone by someone who has repository access but not filesystem access to that server.

That is not nothing. It defends against a leaked clone, a mirror, a misconfigured Git host, a laptop backup of a checkout. Those are real and they happen. But it is a much narrower protection than “our secrets are encrypted” suggests, and the gap between the belief and the reality is where incidents live.

Ask the question in the form that has an answer

Not “are our secrets encrypted”. Ask:

Who can read the vault password, and how would I find out if that set changed?

Then go and check, rather than reasoning about it:

ls -l "$(ansible-config dump | grep VAULT_PASSWORD_FILE | cut -d= -f2- | xargs)"
getent group $(stat -c %G /srv/automation/.vault-pass)

The answer is frequently “every account on this machine”, and almost as frequently “everyone in the developers group, which has forty members and last had someone removed from it in 2023”.

A password file at mode 0600 owned by a dedicated automation account, on a machine only the platform team can log into, is a genuinely different situation from the same file at 0644 on a shared jump box. Both are “a vault password file”. Only one of them is a control.

What the setting actually accepts

Read-only / Safethe documented behaviour— From ansible-config list on ansible-core 2.21.3.

$ ansible-config list | grep -A3 DEFAULT_VAULT_PASSWORD_FILE

DEFAULT_VAULT_PASSWORD_FILE:
description:
- The vault password file to use. Equivalent to --vault-password-file or --vault-id.
- If executable, it will be run and the resulting stdout will be used as the password.
default: null

That second line is the interesting one. If the file is executable, Ansible runs it and uses its standard output. The “password file” is therefore a general extension point, not just a place to keep a string.

Which opens the good options:

Source	Password at rest on the controller?	Auditable centrally?
`prompt`	no	no
plain file	yes	no
executable script	no, if it fetches	yes, if the store logs
client script	no, if it fetches	yes, and it knows the label
keyring via a script	in the keyring, unlocked per session	partly

Executable, and vault-ID-aware

There are two kinds of executable source and the difference is the filename. Measured on 2.21.3:

Read-only / Safea plain executable gets no arguments— Executed on 2.21.3. The script echoes its own argument list to stderr. The list is empty.

$ ansible-vault view --vault-id prod@plainexec.sh p2.yml

invoked with: []

Read-only / Safea script named *-client is told which label is wanted— Executed on 2.21.3. Same script, renamed. Ansible passes --vault-id and the label.

$ ansible-vault view --vault-id dev@prod-client.sh p2.yml

client invoked with: [--vault-id dev]
no password configured for vault-id: dev

A script whose name ends in -client — or -client.EXTENSION — is invoked with --vault-id <label>. A plain executable is invoked with nothing and cannot know which of several passwords is being asked for.

Ansible even distinguishes them in its error text, which is a useful diagnostic when a script is not being called the way you expected:

Vault password script ... returned non-zero (1)
Vault password client script ... returned non-zero (1) when getting secret for vault-id=dev

If you see the first message and expected the second, your script is not named -client.

Read-only / Safea client script that fails closed

#!/bin/sh
# /usr/local/lib/ansible/vault-client.sh
# Named *-client so Ansible passes --vault-id <label>.
# Fetches from the platform secret store; nothing is stored on this host.
set -eu

label=""
while [ $# -gt 0 ]; do
case "$1" in
  --vault-id) label="$2"; shift 2 ;;
  *) shift ;;
esac
done

case "$label" in
dev|staging|prod) ;;
*) echo "refusing unknown vault-id" >&2; exit 1 ;;
esac

# Writes ONLY the password to stdout. Diagnostics go to stderr, and
# must never contain the password - see the warning below.
secret-store get "ansible/vault/$label" 2>/dev/null

Two design points in that script, both load-bearing.

It fails closed. An unrecognised label exits non-zero rather than falling through to a default. A client script that returns the production password for any label it does not recognise is worse than no scheme at all, because it looks like a scheme.

It writes only the password to standard output. Anything else on stdout becomes part of the password — a debug line, a trailing banner, a warning from a tool you called. This is the most common bug in a hand-written client script and it presents as a decryption failure with a correct password, which is a miserable thing to debug.

Your script's stderr ends up in Ansible's error output

Look again at the executed capture above. The script wrote diagnostics to stderr, and Ansible surfaced them verbatim in its own error message:

[ERROR]: Vault password client script ... returned non-zero (1) when
getting secret for vault-id=dev: b'client invoked with: [--vault-id dev]\\n
no password configured for vault-id: dev\\n'

That is good behaviour — it is how you debug the script at all. It also means anything your script writes to stderr can land in a run log, a CI job output, or a pasted terminal session.

So: never write the password, or anything derived from it, to stderr, not even while developing. Do not set -x in a client script. Do not echo the value you just fetched to check it. The moment the script exits non-zero for an unrelated reason, that diagnostic goes into Ansible’s error text and travels wherever that text travels.

Ranking the options

prompt — nothing at rest anywhere. Correct for interactive work against production, and impossible for unattended runs. Use it as the default for a workstation, and stop treating “but I have to type it” as a problem to engineer away: for production material, being unable to run unattended from a laptop is the feature.

Plain file — the common answer. Acceptable when, and only when, the file is mode 0600, owned by a dedicated automation account, on a machine whose login access is itself the control. Its weaknesses are that it does not expire, its access is not logged, and revoking it means finding every copy.

Client script fetching from a secret store — the best of these for an unattended controller. Nothing at rest, access is logged and revocable centrally, and one script serves every label. The cost is a hard dependency on the store, and the same trade-off appears again in the external secret managers lesson.

Keyring — a good fit for an engineer’s workstation. The password lives in the desktop keyring, unlocked once per session by the user’s login. A small script reading from it is an executable password source like any other. It is not a fit for a headless controller, where there is nobody to unlock it.

Reviewing a controller

# What is configured, and does the file exist?
ansible-config dump --only-changed | grep -i vault
grep -rn 'vault_password_file\|vault_identity_list' ansible.cfg ~/.ansible.cfg 2>/dev/null

# Who can read each source?
ls -l ~/.vault/ /srv/automation/.vault-pass 2>/dev/null

# Is a password file inside the repository directory?
git -C /srv/automation status --ignored --short | grep -i vault

# Is one in the environment, where ps and /proc expose it?
env | grep -i vault

That last check finds a pattern worth knowing about: a CI job that writes the vault password into an environment variable, or into a file in the workspace, for the duration of the build. Both are readable by any process in the job, both frequently survive into build logs when something dumps the environment for debugging, and the workspace copy outlives the job if the runner caches the directory.

The --ignored flag on the third command matters. A password file in .gitignore does not show up in a normal git status, which is precisely why people forget it is there.

Under the hood

Why does Ansible not just support “a secret manager” directly, instead of this executable convention?

Because the convention is more general than any integration would be. The contract is small — be executable, print the password on stdout, exit zero, and accept --vault-id if you are named -client — and it composes with anything: HashiCorp Vault, a cloud secret manager, a hardware token, a keyring, a smart card, an internal HTTP service, pass, a script that prompts a human on a different machine.

It also puts the trust boundary in a sensible place. Ansible never holds the credential that fetches the password; the script does, using whatever ambient identity the controller already has — an instance role, a machine certificate, a Kerberos ticket, a systemd credential. That identity is managed by the platform rather than by Ansible, which is where it belongs.

The practical consequence for design: the vault password stops being a thing you distribute and becomes a thing a machine can request. Which changes the offboarding question from “who has a copy” — unanswerable — to “which identities are authorised”, which a secret store answers directly and can revoke without touching any host.

What happens if the controller is compromised while a run is in progress?

Assume the worst honestly, because the answer determines what the controls in this part were ever worth.

An attacker with the automation account on the controller, during a run, has:

the vault password, or the ability to obtain it the same way Ansible just did — the client script runs as that account, so it will serve them too
every decrypted secret in the process memory of the running play
the SSH key or agent socket that reaches the fleet
the inventory, which is a map of what to attack next

No configuration in this part prevents that. Vault protects data at rest in a repository. It does not protect a machine that is executing your automation from someone who controls that machine.

What the controls in this part change is the scope and the aftermath:

A client script leaves an audit trail. The secret store recorded that a password was fetched, by which identity, at what time. A password file records nothing, so you cannot tell whether the compromise touched secrets at all.
Revocation is possible and fast. Withdraw the controller’s identity at the store and the vault password stops being obtainable everywhere at once, without logging into anything. A distributed password file has to be hunted.
Vault IDs bound the blast radius. A controller that only ever held the staging identity did not have production material to lose, whatever else it lost.

The design principle this leads to, and it is the one that survives every other detail in this part: a controller that can decrypt production secrets is a production system. Give it the monitoring, the access control, the patching and the change process that any production system gets. Most estates treat the controller as a tool, and it is the machine with the most reach in the entire environment.

Knowledge check

Why this matters

The vault password is the most valuable secret in the estate and it gets the least design attention, because by the time you are choosing where to put it the interesting work — encrypting things — already feels done.

The result is a file in the repository directory, mode 0644 because that is what the umask produced, on a machine several teams can log into, with no record of who has copied it and no way to withdraw it. And the estate describes itself, accurately in one sense and misleadingly in every other, as having encrypted secrets.

The measurable improvement is not exotic. Move the password out of the filesystem and behind an executable that fetches it, and three things change at once: nothing sits at rest to be copied, every request is logged, and revocation becomes one action in one place. The contract for doing that is four lines long, and this lesson measured every part of it.