Runbook: Rotate Vault credentials

Rotating a vault password is a repository-wide, all-or-nothing change with a distributed set of consumers. The failure is not dramatic: the rekey succeeds, the merge lands, and then one CI runner and two operators discover at different times over the next fortnight that they can no longer decrypt anything.

Read the first callout before anything else. It decides whether this is the right runbook.

When to use this runbook

• Scheduled vault password rotation.
• A person with the password has left the team.
• The password was handled carelessly and needs replacing.
• Splitting a single password into per-environment vault IDs (the rekey mechanics are the same; the target ID differs per file).

Blast radius

Every consumer of the password, which is more than the fleet:

• Every controller with a vault password file.
• Every CI runner and every pipeline secret.
• Every operator who runs playbooks by hand.
• Every scheduled job, timer or platform credential.
• Every stale clone on someone’s laptop.

Managed hosts are not in the blast radius. Nothing changes on them. What breaks is the ability to run automation at all, which is worse in a different way: you find out during the next incident.

Inputs

• The current vault password, from the password manager.
• The list of encrypted files and their vault IDs.
• The consumer list.
• A merge freeze window.

Step 1: Freeze and enumerate

cd /srv/automation/repo
grep -rl --binary-files=without-match '^\$ANSIBLE_VAULT' . \
--exclude-dir=.git | sort | tee vault-files.txt
wc -l vault-files.txt

Then read the header of each one, because the header carries the vault ID:

while read -r f; do
printf '%-60s %s\n' "$f" "$(head -1 "$f")"
done < vault-files.txt

A 1.2 format header ends with the vault ID label:

inventories/production/group_vars/all/vault.yml   $ANSIBLE_VAULT;1.2;AES256;prod
inventories/staging/group_vars/all/vault.yml      $ANSIBLE_VAULT;1.2;AES256;dev

A 1.1 header has no label, which means the file was encrypted without a vault ID and any configured password will be tried against it. Note those separately - they are the files most likely to be missed by an ID-scoped rekey.

Step 2: Prove the current password works everywhere

rc=0
while read -r f; do
if ansible-vault view --vault-id prod@/home/ansible/.vault-pass "$f" >/dev/null 2>&1; then
  echo "ok      $f"
else
  echo "FAILED  $f"
  rc=1
fi
done < vault-files.txt
exit $rc

Any FAILED line here is a file that is already orphaned - encrypted under a password nobody has, or under a different vault ID. Resolve that before rotating. Rekeying cannot fix a file you cannot open, and after the rotation you will have two passwords that do not work on it instead of one.

Step 3: Generate and store the new password first

python3 -c 'import secrets; print(secrets.token_urlsafe(32))' \
> /home/ansible/.vault-pass-new
chmod 600 /home/ansible/.vault-pass-new

Put it in the password manager before you use it. A password that exists only in a file on one controller, and is then used to rekey the whole repository, is one disk failure away from making every encrypted file in your automation unreadable.

Do not type it on a command line. ansible-vault takes --vault-password-file and --vault-id label@file precisely so that the value never appears in the process table or in shell history.

Step 4: Rekey on a branch

git checkout -b chore/vault-rekey-2026-08
git status --short

while read -r f; do
ansible-vault rekey \
  --vault-id prod@/home/ansible/.vault-pass \
  --new-vault-id prod@/home/ansible/.vault-pass-new \
  "$f"
done < vault-files.txt

Verified on 2.21.3: rekey prints Rekey successful per file, keeps the vault ID label in the header ($ANSIBLE_VAULT;1.2;AES256;prod), and rewrites the file in place.

If you are moving a file to a different vault ID at the same time, that is the --new-vault-id label, and the header changes accordingly. Do one thing at a time: a rotation that also reorganises IDs makes the “which files did I miss” question much harder to answer.

Step 5: Verify - new works, old fails, content unchanged

Three checks. All three matter, and the second one is the one people leave out.

while read -r f; do
ansible-vault view --vault-id prod@/home/ansible/.vault-pass-new "$f" >/dev/null \
  && echo "new ok   $f" || echo "NEW FAILED $f"
done < vault-files.txt

while read -r f; do
if ansible-vault view --vault-id prod@/home/ansible/.vault-pass "$f" >/dev/null 2>&1; then
  echo "STILL OPENS WITH OLD PASSWORD: $f"
else
  echo "rekeyed ok                     $f"
fi
done < vault-files.txt

Verified failure message on 2.21.3, which is what you should expect to see suppressed above:

Decryption failed (no vault secrets were found that could decrypt).

A file that still opens with the old password was not rekeyed - most often because it was added to the repository after vault-files.txt was generated, which is what the merge freeze exists to prevent.

while read -r f; do
git show "HEAD:$f" > /tmp/old.vault
a=$(ansible-vault view --vault-id prod@/home/ansible/.vault-pass /tmp/old.vault | sha256sum)
b=$(ansible-vault view --vault-id prod@/home/ansible/.vault-pass-new "$f" | sha256sum)
[ "$a" = "$b" ] && echo "identical $f" || echo "CONTENT CHANGED $f"
rm -f /tmp/old.vault
done < vault-files.txt

This is the check that catches an editor that reformatted a file, a truncated write, or a rekey that ran against a partially written file. CONTENT CHANGED on any line stops the rotation.

Step 6: Distribute before you merge

Work down the consumer list from the pre-checks. For each one, the new password must be in place before the merge, because the moment the rekeyed files land on the main branch, anything still holding the old password is broken.

• Controllers - write the new password file, keep the old one alongside temporarily.
• CI - update the pipeline secret; do not rely on a cached value.
• Operators - the password manager entry, and a note that they must refresh their local password file.
• Scheduled jobs and platform credentials - these are the ones that get forgotten, because nobody is watching when they run.

A controller can hold both during the transition, which removes the timing pressure entirely:

[defaults]
vault_identity_list = prod@/home/ansible/.vault-pass-new, prod_old@/home/ansible/.vault-pass

With both listed, files encrypted under either password decrypt. That is a transition aid and a security weakness at the same time, so it is a line in the change record with a removal date, not a permanent configuration.

Step 7: Merge, then prove it from a controller

git push -u origin chore/vault-rekey-2026-08
# open the merge request, get it approved, merge

Immediately after the merge, from each controller:

cd /srv/automation/repo && git pull --ff-only
ansible-playbook site.yml --limit staging-web01.example.com --check
echo "exit=$?"

Exit code 0 and no decryption error. A vault failure surfaces as a task-level error mentioning decryption, and it fails at the point the variable is first needed - which can be several tasks in, not at startup.

Step 8: Retire the old password

# On every controller
shred -u /home/ansible/.vault-pass
mv /home/ansible/.vault-pass-new /home/ansible/.vault-pass

# Remove the transitional dual-identity line from ansible.cfg
# Archive, then delete, the old password manager entry

Also remove the transitional vault_identity_list entry. A controller left holding both passwords indefinitely means the rotation achieved nothing: the old credential still opens everything.

Step 9: If this was exposure, rotate the contents

Re-read the callout at the top. If the password leaked, the plaintext leaked. Enumerate what the vault contained and rotate each item through its own procedure - database passwords, API tokens, certificates, service account keys. That work is tracked separately and it is the part that actually closes the exposure.

Rollback

while read -r f; do
ansible-vault rekey \
  --vault-id prod@/home/ansible/.vault-pass-new \
  --new-vault-id prod@/home/ansible/.vault-pass \
  "$f"
done < vault-files.txt

Common patterns

Escalation

Escalate when:

• The rotation is a response to exposure. The secrets inside need rotating and that is a security-owned workstream.
• A file cannot be decrypted with the current password before you start.
• CI cannot be updated inside the window.
• Encrypted files exist on branches with no owner, and destroying the old password would strand them.