Lab: A vault-ID split that actually holds

Objective

By the end of this lab you will have two encrypted files under two vault identities and will have run four decryption attempts that establish exactly what the identity label does and does not guarantee. The headline result is one most people get wrong: by default, the label on a vault file is documentation, and Ansible will happily try every password it has been given against every file regardless of label.

Architecture

One repository, two environments, two vault identities, two password sources.

repo/
├── inventories/
│   ├── dev/group_vars/all/vault.yml    $ANSIBLE_VAULT;1.2;AES256;dev
│   └── prod/group_vars/all/vault.yml   $ANSIBLE_VAULT;1.2;AES256;prod
├── .vault-dev      <- password file, mode 0600
└── .vault-prod     <- password file, mode 0600

Requirements

A controller with ansible-core 2.21.x. All output below was captured from 2.21.3.
No managed nodes and no network access. Every operation is ansible-vault on local files, plus one debug play against ansible_connection: local.
Somewhere to write two password files. They will be created mode 0600 and deleted in Cleanup.

Scenario

Your repository holds credentials for three environments. Everyone on the team has the dev vault password because everyone runs dev locally. Two people have the production password.

Somebody asks the reasonable question: “if I run a playbook with both passwords loaded, what stops a mistake in the inventory path from decrypting the production credentials into a dev run?” Nobody knows. Your job is to find out empirically and then to configure whatever it takes to make the answer be “the label does”.

Tasks

Task 1: Set up the identities

WORKDIR="$HOME/ansible-vaultid-lab"
mkdir -p "$WORKDIR"/inventories/{dev,prod}/group_vars/all
cd "$WORKDIR"

Create the two password files. Mode 0600 first, content second — never the other way round:

umask 077

printf 'lab-dev-password-REPLACE_ME\n'  > .vault-dev
printf 'lab-prod-password-REPLACE_ME\n' > .vault-prod

chmod 0600 .vault-dev .vault-prod
ls -l .vault-dev .vault-prod

Write the plaintext material, then encrypt it:

cat > inventories/dev/group_vars/all/vault.yml <<'YAML'
vault_db_password: dev-REDACTED
vault_api_token: dev-token-REPLACE_ME
YAML

cat > inventories/prod/group_vars/all/vault.yml <<'YAML'
vault_db_password: prod-REDACTED
vault_api_token: prod-token-REPLACE_ME
YAML

ansible-vault encrypt --vault-id dev@.vault-dev   inventories/dev/group_vars/all/vault.yml
ansible-vault encrypt --vault-id prod@.vault-prod inventories/prod/group_vars/all/vault.yml

Task 2: Read the label from the file

The identity is recorded in the file header, in plaintext:

Read-only / Safecontroller— The vault header. Format version, cipher, and the identity label.

$ head -1 inventories/dev/group_vars/all/vault.yml inventories/prod/group_vars/all/vault.yml

==> inventories/dev/group_vars/all/vault.yml <==
$ANSIBLE_VAULT;1.2;AES256;dev

==> inventories/prod/group_vars/all/vault.yml <==
$ANSIBLE_VAULT;1.2;AES256;prod

Format 1.2 is what carries a label; 1.1 files have no identity field. The label is dev and prod respectively, and it is readable by anyone who can read the file — which is the point. It tells a tool which password to reach for. It is not a secret and it is not, by itself, a control.

Task 3: The four decryption attempts

Run all four and record each result. The pattern is the lab.

Attempt 1 — correct identity, correct password.

ansible-vault view --vault-id prod@.vault-prod \
  inventories/prod/group_vars/all/vault.yml

Succeeds, as expected.

Attempt 2 — dev identity only, against the prod file.

Read-only / Safecontroller— The dev password cannot open the prod file

$ ansible-vault view --vault-id dev@.vault-dev inventories/prod/group_vars/all/vault.yml

Decryption failed (no vault secrets were found that could decrypt).
Origin: /home/operator/ansible-vaultid-lab/inventories/prod/group_vars/all/vault.yml

Good. But note why it failed: the only password supplied was the wrong one. The label was not consulted at all.

Attempt 3 — both identities supplied, against the prod file. This is the case in the scenario: an operator who has both passwords loaded.

Read-only / Safecontroller— Both passwords loaded. The prod file opens.

$ ansible-vault view --vault-id dev@.vault-dev --vault-id prod@.vault-prod inventories/prod/group_vars/all/vault.yml

vault_db_password: prod-REDACTED
vault_api_token: prod-token-REPLACE_ME

Expected, and correct: the prod password was supplied, so the prod file opened. Still no information about whether the label did anything.

Attempt 4 — the one that answers the question. Supply the dev password under a wrong label, against the dev file:

Read-only / Safecontroller— Label says 'anything'. File says 'dev'. It decrypts anyway.

$ ansible-vault view --vault-id anything@.vault-dev inventories/dev/group_vars/all/vault.yml

vault_db_password: dev-REDACTED
vault_api_token: dev-token-REPLACE_ME

Task 4: Turn the label into a boundary

; ansible.cfg
[defaults]
vault_id_match = True

Or, for a single invocation:

export ANSIBLE_VAULT_ID_MATCH=True

Re-run attempt 4:

Read-only / Safecontroller— Same password, same file, matching now enforced

$ ANSIBLE_VAULT_ID_MATCH=True ansible-vault view --vault-id anything@.vault-dev inventories/dev/group_vars/all/vault.yml

Decryption failed (no vault secrets were found that could decrypt).
Origin: /home/operator/ansible-vaultid-lab/inventories/dev/group_vars/all/vault.yml

Refused. The password was right and the label was wrong, and with matching on, that is enough.

Confirm the correct label still works:

ANSIBLE_VAULT_ID_MATCH=True ansible-vault view --vault-id dev@.vault-dev \
  inventories/dev/group_vars/all/vault.yml

Task 5: Encrypt a single string, not a whole file

Whole-file encryption makes a variables file unreviewable — every change is a diff of ciphertext. encrypt_string encrypts one value, leaving the rest of the file readable:

ansible-vault encrypt_string --vault-id prod@.vault-prod \
  --stdin-name 'vault_smtp_password' <<'EOF'
prod-smtp-REDACTED
EOF

Paste the output into a plain, unencrypted group_vars file:

# inventories/prod/group_vars/all/mail.yml  (NOT encrypted as a whole)
smtp_host: smtp.example.com
smtp_port: 587
smtp_user: mailer

vault_smtp_password: !vault |
  $ANSIBLE_VAULT;1.2;AES256;prod
  35363...
  ...

Now a reviewer can see that the SMTP host changed without needing the vault password, and only the secret itself is opaque.

The convention that makes this work

Prefix every encrypted variable with vault_, and never consume a vault_-prefixed variable directly in a template or task. Instead, map it in a plain file:

# inventories/prod/group_vars/all/mail.yml
smtp_password: "{{ vault_smtp_password }}"

Two reasons. It makes every secret in the repository greppable by name. And it means the role or template consumes a normally-named variable, so the role does not need to know that the value happens to be encrypted — which is what lets the same role run in an environment that supplies the value from a secret manager instead.

Task 6: Rekey, and prove the old password is dead

Rotation is the operation that separates a vault you can manage from one you are stuck with. Rekeying changes the password and can change the identity label:

umask 077
printf 'lab-prod-password-v2-REPLACE_ME\n' > .vault-prod-v2
chmod 0600 .vault-prod-v2

ansible-vault rekey \
  --vault-id prod@.vault-prod \
  --new-vault-id prod@.vault-prod-v2 \
  inventories/prod/group_vars/all/vault.yml

Prove both halves:

# The new password opens it
ansible-vault view --vault-id prod@.vault-prod-v2 \
  inventories/prod/group_vars/all/vault.yml

# The old password does not
ansible-vault view --vault-id prod@.vault-prod \
  inventories/prod/group_vars/all/vault.yml

The second must fail with Decryption failed.

Rekey does not rotate the secret, only the lock

The file now has a new password. The database password inside it is unchanged, and anyone who read it while holding the old vault password still knows it.

A vault rekey is the right response to “the vault password may have leaked”. It is not a response to “the database password may have leaked” — that requires changing the database password, then re-encrypting, in that order. Confusing the two produces an incident report that says the credential was rotated when it was not.

encrypt_string values are rekeyed individually, and ansible-vault rekey does not find them: they live inside otherwise-plain YAML files that the tool does not treat as vaults. Rotating a vault identity across a real repository therefore means finding every !vault | block as well:

grep -rln '\$ANSIBLE_VAULT' . --include='*.yml' --include='*.yaml'

Task 7: Use it from a playbook

# show.yml
- name: Show which environment's secret is in scope
  hosts: all
  gather_facts: false
  tasks:
    - name: Report the loaded secret, redacted
      ansible.builtin.debug:
        msg: "db password ends with ...{{ vault_db_password[-8:] }}"
      no_log: false

# inventories/dev/hosts.yml
all:
  hosts:
    devnode:
  vars:
    ansible_connection: local

export ANSIBLE_VAULT_ID_MATCH=True

ansible-playbook -i inventories/dev/hosts.yml show.yml \
  --vault-id dev@.vault-dev

# Now try to run the dev inventory with only the prod identity
ansible-playbook -i inventories/dev/hosts.yml show.yml \
  --vault-id prod@.vault-prod

Read-only / Safecontroller— Wrong identity for this inventory. Fails before the first task.

$ ansible-playbook -i inventories/dev/hosts.yml show.yml --vault-id prod@.vault-prod; echo "exit=$?"

PLAY [all] *********************************************************************
[ERROR]: Decryption failed (no vault secrets were found that could decrypt).
exit=4

The play banner prints — the variables are resolved lazily, at first use — but no task runs and the exit code is 4, Ansible’s parse/setup error. That is the behaviour you want from a wrong-environment invocation: it stops before touching anything, and the exit code is distinguishable from a task failure (exit 2).

Validation

head -1 on each vault file shows $ANSIBLE_VAULT;1.2;AES256;dev and ...;prod respectively.
With --vault-id dev@.vault-dev alone, the prod file fails to decrypt.
With both identities supplied, the prod file decrypts.
With ANSIBLE_VAULT_ID_MATCH unset, --vault-id anything@.vault-dev decrypts the dev file.
With ANSIBLE_VAULT_ID_MATCH=True, the same command fails, and --vault-id dev@.vault-dev still succeeds.
After the rekey, prod@.vault-prod-v2 opens the prod file and prod@.vault-prod does not.
ls -l .vault-* shows mode -rw------- on every password file.
ansible-playbook -i inventories/dev/hosts.yml show.yml --vault-id prod@.vault-prod prints the play banner, fails with Decryption failed before any task runs, and exits 4.

Expected Outcome

ansible-vaultid-lab/
├── ansible.cfg                 vault_id_match = True
├── .vault-dev, .vault-prod, .vault-prod-v2   (mode 0600)
├── attempts.md                 the four results
├── inventories/
│   ├── dev/{hosts.yml, group_vars/all/vault.yml}
│   └── prod/group_vars/all/{vault.yml, mail.yml}
└── show.yml

Two environments, two identities, matching enforced, one identity rotated and verified. You can state precisely what the label guarantees with matching off (nothing) and with it on (that a mistake fails rather than succeeding against the wrong environment).

Troubleshooting

ERROR! Attempting to decrypt but no vault secrets found. No --vault-id, no --vault-password-file, and no DEFAULT_VAULT_IDENTITY_LIST in config. Ansible has nothing to try.

The password file “works” but produces a wrong password. A trailing newline is stripped, but any other whitespace is not. echo -n versus printf versus an editor that adds a final newline all produce different files. If a password file that looks right fails, xxd .vault-dev | tail -2.

An executable password file behaves unexpectedly. If the file has the execute bit set, Ansible runs it and uses its stdout as the password. That is a feature — it is how a script can fetch from a secret manager — and a surprise when a chmod -R +x made your password file executable.

vault_id_match seems to have no effect. Confirm it is actually in effect: ansible-config dump --only-changed | grep -i vault. An ansible.cfg in the wrong directory, or one skipped for being in a world-writable directory, is the usual cause.

A rekeyed file cannot be opened by anyone. ansible-vault rekey rewrites in place. If it was interrupted, the file may be truncated. Restore from version control — which is the reason encrypted files belong in git and password files never do.

encrypt_string output pasted into YAML fails to parse. The !vault | tag and the indentation must both be right. Let encrypt_string produce the whole block and paste it verbatim; do not retype the indentation.

Cleanup

This lab created password files and encrypted material inside one directory, and may have set an environment variable in your shell.

Step 1. Unset anything exported into the current shell:

unset ANSIBLE_VAULT_ID_MATCH
env | grep -i vault || echo 'no vault environment variables set'

Step 2. Confirm no password file escaped the working directory. This is the check that matters:

cd "$HOME/ansible-vaultid-lab"
ls -la .vault-* 2>/dev/null

# Anything vault-shaped elsewhere in your home directory?
find "$HOME" -maxdepth 2 -name '.vault-*' -not -path "$HOME/ansible-vaultid-lab/*" 2>/dev/null

Step 3. Overwrite the password files before deleting them. rm unlinks; it does not erase, and on a journalling filesystem the content can persist:

cd "$HOME/ansible-vaultid-lab"

for f in .vault-dev .vault-prod .vault-prod-v2; do
  [ -f "$f" ] && dd if=/dev/urandom of="$f" bs=64 count=1 conv=notrunc status=none
done

shred -u .vault-dev .vault-prod .vault-prod-v2 2>/dev/null \
  || rm -f .vault-dev .vault-prod .vault-prod-v2

Step 4. Keep the notes, remove the directory:

rm -rf deletes the whole lab directory, including anything you added

The next block removes ~/ansible-vaultid-lab and everything under it, recursively and without confirmation. If you saved notes, an inventory you want to keep, or a playbook you extended, copy it out first — the step above does that for the lab’s own deliverables and knows nothing about yours.

The path is written in full and anchored to $HOME deliberately. A relative rm -rf run from the wrong directory is how people delete work they meant to keep, and the shell gives no second chance.

Confirm what you are about to delete before you delete it:

ls -la "$HOME/ansible-vaultid-lab"

mkdir -p "$HOME/ansible-lab-deliverables"
cp -a "$HOME/ansible-vaultid-lab/attempts.md" \
      "$HOME/ansible-lab-deliverables/vault-id-attempts.md"

rm -rf "$HOME/ansible-vaultid-lab"

Step 5. Check your shell history for anything that should not be there:

history | grep -i -E 'vault|password' | tail -20

A password typed at a --ask-vault-pass prompt is not in history. One passed on a command line is, and is also visible in ps output to every user on the machine for the life of the process. That is why every example in this lab used a file.

Under the hood

A vault-encrypted payload starts with a plaintext header line:

$ANSIBLE_VAULT;1.2;AES256;dev

Four semicolon-separated fields: the magic string, the format version, the cipher, and — in 1.2 only — the identity label. Everything after the header is hex-encoded ciphertext with an HMAC.

When Ansible meets a vault, it assembles the list of secrets it has been given — from --vault-id, --vault-password-file, --ask-vault-pass and DEFAULT_VAULT_IDENTITY_LIST — and tries them. With vault_id_match false, it tries them all, in the order supplied, and the first whose HMAC verifies wins. The label in the header is used only as an ordering hint: a secret whose label matches is tried first.

With vault_id_match true, the list is filtered to secrets whose label equals the header’s before anything is attempted. A file whose label matches nothing you hold fails immediately rather than after a fruitless sweep.

This also explains the performance note that occasionally surprises people: each attempt runs a full key derivation, which is deliberately slow. A repository with many vault files and many loaded identities, with matching off, does a lot of pointless PBKDF2 work at inventory load. Turning matching on is a correctness improvement that happens to also be faster.

What You Learned

The identity label is in the file header, in plaintext. It is metadata for tooling, not a secret.
With vault_id_match off — the default — a wrong label still decrypts if the password is right. You proved it with --vault-id anything@.vault-dev against a file labelled dev.
With matching on, the same command fails. The label becomes a boundary, and a wrong-environment invocation fails at inventory load instead of running.
What keeps production out of a dev run is not holding the production password. Matching turns a mistake into an error; it does not turn a held password into a non-held one.
encrypt_string keeps a variables file reviewable, and the vault_-prefix-then-map convention keeps roles ignorant of whether a value happens to be encrypted.
Rekey rotates the lock, not the secret. A leaked database password needs the database password changed; a leaked vault password needs a rekey. They are different incidents.

Objectives

Prerequisites