Break/Fix: after the vault key rotation, production fails to decrypt and staging is fine · Break/Fix · Ansible

Reported symptoms

A vault key rotation was completed on Friday afternoon. On Monday morning every play that targets production fails before running a single task.

PLAY [production] **************************************************************
[ERROR]: Decryption failed (no vault secrets were found that could decrypt).

That is the whole message. It does not name a file, so the first job is to work out which of the repository vaulted artefacts it is talking about.

Staging is entirely unaffected and has been running normally all weekend.

The engineer who performed the rotation checks their work and finds nothing wrong:

The new password file is in place on the controller and is readable.
ansible-vault view on the failing file, with the new password, displays the contents correctly.
The rotation command reported success on every file it processed.
No file is corrupt.

So the file can be decrypted by hand with the password the run has, and the run cannot decrypt it. That is where the morning goes.

Evidence provided

Read-only / Safefour fields - the last one is a label

$ head -1 group_vars/production/vault.yml

$ANSIBLE_VAULT;1.2;AES256;prod2026

Read-only / Safethree fields - no label, and this one still works

$ head -1 group_vars/staging/vault.yml

$ANSIBLE_VAULT;1.1;AES256

Read-only / Safethe run has exactly one secret, and it is the old one

$ ansible-config dump | grep -i vault

DEFAULT_VAULT_IDENTITY_LIST(/srv/automation/ansible.cfg) = ['default@/etc/ansible/vault-pass-old']
DEFAULT_VAULT_PASSWORD_FILE(default) = None

Read-only / Safeby hand, with the right identity, it opens perfectly

$ ansible-vault view --vault-id prod2026@/etc/ansible/vault-pass-new group_vars/production/vault.yml | head -1

db_password: REDACTED

Read-only / Safefive vaulted artefacts - the rotation covered group_vars only

$ grep -rl 'ANSIBLE_VAULT' . | head

./group_vars/production/vault.yml
./group_vars/staging/vault.yml
./host_vars/db01/vault.yml
./roles/monitoring/vars/secrets.yml
./roles/backup/files/credentials.enc

Read-only / Safeone rotated, four not - and only one of them fails loudly

$ grep -rl ANSIBLE_VAULT . | while read -r f; do echo "$(head -1 "$f" | cut -d';' -f4) $f"; done

 ./group_vars/staging/vault.yml
prod2026 ./group_vars/production/vault.yml
./host_vars/db01/vault.yml
./roles/monitoring/vars/secrets.yml
./roles/backup/files/credentials.enc

Work the evidence before reading on

The file decrypts by hand and does not decrypt in the run. The password is the same in both cases, so the password is not the difference.

Compare the two vault headers field by field. How many fields does each have, and what is the extra one?
Compare the label in the failing header against the identity list the configuration provides. Do they name the same thing?
Count the vaulted files the grep found, and count the ones the rotation touched.

Before continuing: staging still works. Is that because staging was rotated correctly, or because it was not rotated at all?

Root cause

1. The vault header records the identity a file was encrypted under

A vaulted file begins with a single header line:

$ANSIBLE_VAULT;1.1;AES256
$ANSIBLE_VAULT;1.2;AES256;prod2026

Format 1.1 has three fields. Format 1.2 adds a fourth, the vault identity label, which records which named secret the file was encrypted with.

The rotation used --vault-id prod2026@..., so the production file was re-encrypted under the label prod2026 and its header records that.

2. The run was given one secret and it is the old one

DEFAULT_VAULT_IDENTITY_LIST provides the secrets available to a run. On this controller it names one entry: the old password file, under the label default.

Ansible tries the secrets it has. It does not have the new password at all, under any label, so nothing can decrypt the production file and it refuses with no vault secrets were found that could decrypt.

The by-hand test succeeded because it supplied the new identity explicitly on the command line. It was a test of the file, not of the configuration, and it therefore confirmed the wrong thing.

3. The rotation missed four of five vaulted artefacts

The rotation command was pointed at group_vars/. Vaulted content also lives in host_vars/, in role vars/, and in files bundled with roles - and can live as individual encrypted strings inline in an otherwise plain YAML file, where no filename gives it away.

Four artefacts still carry the old, unlabelled header. They keep working precisely because they were missed, which is why staging appeared healthy and gave a false sense that the rotation had gone well.

Under the hood

Ansible tries every secret it holds against every vaulted payload, and the label is a hint rather than a lock. That has two consequences worth knowing.

The first is forgiving. A file labelled prod2026 will decrypt under a secret supplied with a different label, as long as the password is right. Mislabelling is therefore recoverable and does not corrupt anything.

The second is that a run configured with several identities can read a repository that is halfway through a rotation:

[defaults]
vault_identity_list = old@/etc/ansible/vault-pass-old, prod2026@/etc/ansible/vault-pass-new

That is the intended way to run during a migration, and it is what makes an incremental rotation possible. Without it a rotation has to be atomic across every vaulted artefact in the repository, which is exactly the requirement this rotation failed to meet.

It also explains why the failure was so abrupt. Decryption of variable files happens while variables are being loaded, before the first task, so the whole play fails at once rather than partway through. That is the good version of the failure - nothing was partially applied - and it is the reason this incident is a morning rather than a recovery.

Enumerate vaulted content by content, not by path

The rotation was run against group_vars/ because that is where secrets are supposed to live. Four of the five vaulted artefacts were somewhere else, and none of them was in an unreasonable place.

Vaulted content is identifiable by its first line regardless of where it sits, so the enumeration is one command:

grep -rl 'ANSIBLE_VAULT' .

That finds whole encrypted files. Inline encrypted strings - produced by ansible-vault encrypt_string and embedded in an otherwise readable YAML file - carry the same marker inside the file rather than on the first line, so they are found by the same search and are easy to miss when scanning by filename.

Any rotation procedure whose first step is a list of directories is a procedure that will miss something. The first step should produce the list, not consume one.

Resolution

Restore the ability to run by adding both identities to the configuration. This is what labelled identities are for and it is the intended state during a rotation, not a workaround.
Confirm production plays parse and run again before doing anything else, so the estate is operable while the rotation is finished properly.
Enumerate every vaulted artefact by content. grep -rl ANSIBLE_VAULT . across the whole repository, including roles, files bundled with roles, and any inline encrypted strings.
Re-encrypt everything outstanding under the new identity, working from that list rather than from a list of directories.
Verify each file individually after re-encrypting, by checking the header label rather than by trusting the command output.
Remove the old identity from the configuration. This is the step that declares the rotation complete, and until it happens the old key is still valid everywhere.
Run every play with the new identity alone and require them all to parse. A play that only runs because the old key is still accepted has not been migrated.
Establish whether the underlying credentials also need changing. Re-encrypting a value under a new key does not change the value, and if the old key was exposed then so was everything it protected.

Verification

Everything decrypts under the new identity alone. Remove the old identity from the configuration and require every play in the repository to parse successfully. This is the check that proves the rotation finished.
The check can fail. On a scratch branch, leave one file encrypted under the old key and confirm the run refuses. A verification that passes with an unrotated file present is measuring nothing.
The enumeration is complete. Re-run the content search and confirm every result carries the new label in its header.
The old password opens nothing. Attempt to view each artefact with the old password alone and require failure on all of them.
A CI check exists and works. Add a build step that decrypts every vaulted artefact with the configured identities, and confirm it goes red when a file is left behind.
The credentials themselves were handled. If the old vault key was exposed, confirm the secrets it protected have been changed at their source and that the old values no longer authenticate.
Both staging and production are on the same identity. Splitting them was the accident that hid the problem, and leaving them split leaves the next rotation with the same trap.

Prevention

Enumerate vaulted content by searching for the vault header, never by listing directories. The first step of a rotation should produce the list of files, not assume it.
Use labelled vault identities from the start. They are what makes an incremental rotation possible, and without them a rotation has to be atomic across the entire repository.
Run with both identities during the migration window, deliberately and with an end date. The window is the mechanism; leaving it open forever means the old key is still valid and nothing was rotated.
Removing the old identity is what completes the rotation. Until that step, the procedure is unfinished no matter how many files were re-encrypted.
Add a CI check that every vaulted artefact decrypts under the currently configured identities. A file missed by a rotation should fail a build, not a production run at 09:00 on a Monday.
Verify configuration, not just files. Testing a file with an explicit --vault-id on the command line proves the file is fine and says nothing about what the run will be able to open.
Remember that re-encrypting is not rotating. If the old key was exposed, every secret it protected is exposed, and those values have to change at their source.

What if the rotation had produced unlabelled files under a new password instead?

Then the failure would have been the same and the recovery would have been slightly harder, because there would be no label to tell the files apart.

Without labels, every vaulted file has an identical three-field header and the only way to know which key opens which file is to try. A half-rotated repository becomes a set of files that must be classified by experiment rather than read.

Supplying both passwords still works - Ansible tries each secret against each payload - so the estate is still operable. What is lost is the ability to answer “which files still need rotating” without decrypting them, and that is precisely the question a rotation needs answered.

There is a third variant worth planning for, and it is the unrecoverable one. If a rotation re-encrypts files under a new key and the new key is then lost, the contents are gone. ansible-vault rekey decrypts and re-encrypts in place, so there is no copy under the old key afterwards. Any rotation procedure should therefore begin with a verified backup of the repository state and end with a verified copy of the new key in whatever system holds keys - before the old one is retired.

Why this matters

The rotation was performed carefully and verified carefully, and it was verified against the wrong thing. Testing a file with an explicit --vault-id proves the file is intact; it says nothing about whether the configuration a run uses can open it. Those are separate claims and only the second one matters on Monday morning.

The wider point is about enumeration. Secrets management fails at the edges - the file in a role instead of group_vars, the encrypted string inline in a plain YAML file, the credential in a repository nobody listed. A rotation that starts from a list of places will always miss one, and the ones it misses keep working, which reads as success.

Start from a search, finish by removing the old key, and make the check that the old key opens nothing the definition of done.

Reported symptoms

Evidence

Root cause

Remediation

Verification

Prevention

Reported symptoms

Evidence provided

Work the evidence before reading on

Root cause

1. The vault header records the identity a file was encrypted under

2. The run was given one secret and it is the old one

3. The rotation missed four of five vaulted artefacts

Resolution

Verification

Prevention