Backing up the things you must not lose or leak

Every other item in this part has an easy answer: put it in Git. This lesson is about the two items where that answer is forbidden, and where the requirement pulls in two directions at once.

The material must survive the controller’s destruction. The material must not become stealable.

Ordinary backup thinking optimises the first and quietly degrades the second — more copies, more places, more people who can restore. Ordinary security thinking optimises the second and quietly degrades the first — one copy, one holder, minimum exposure. Neither on its own is a plan for a vault password, because a vault password with one copy on a destroyed host is gone, and a vault password with fifteen copies is a published credential.

What has to be escrowed

Two items, plus whatever your environment adds:

Read the two columns together. This is the only category in the estate where losing it and leaking it are both terminal, which is why it gets a lesson rather than a bullet point.

Split custody: nobody can open it alone

The mechanism that satisfies both directions of the requirement is straightforward and old: the material is recoverable by a quorum and by no individual.

Two implementations, both legitimate:

Sealed halves. The password is split into two parts. Each half goes to a different custodian, in a different physical or organisational location. Neither half alone reveals anything useful about the password, and both are needed to reconstitute it. Simple to explain, simple to audit, and it fails if either custodian is unavailable.

Threshold sharing. The password is split into n shares of which any k reconstruct it — three of five, for example. Tolerates absent custodians, which matters at 03:00 on a public holiday, at the cost of being a mechanism your team has to actually understand.

What goes in the envelope

An escrow containing only a password is incomplete. It has to contain enough that someone who has never seen your environment can use it.

1. The secret material itself - vault password per vault ID, and the automation private key.
2. A label saying what each item is for, in words, not just a filename. "Vault password, ID prod, repository git.example.com/ansible/fleet" beats "vault-prod.txt".
3. The date it was placed, and the date the material was last rotated. A mismatch between these two is the failure the break/fix scenario for this part is built on.
4. The git URL of the repository it decrypts, and the branch that is authoritative.
5. The recovery instructions for the escrow mechanism itself, including the tool and its version.
6. The name of the runbook to follow next - the controller rebuild runbook from lesson 6.
7. The names or roles of the custodians and the quorum required, so whoever opens it knows whether they have opened it legitimately.

The seventh item is the one that turns an envelope into a control. An escrow whose opening cannot be distinguished from a theft is not providing much.

Break-glass: the procedure that assumes it is 03:00

Break-glass is the documented path from “the controller is gone” to “I am holding the vault password”, and it has to be written before it is needed because the conditions under which it is used are the worst possible conditions for improvisation.

1. Trigger: state the conditions that justify opening the escrow. "The controller is unrecoverable and a change to the fleet is required." Not "somebody wants the password".
2. Authorisation: who may authorise, and what is the fallback if that person is unreachable. A procedure requiring one named individual is a procedure with a single point of failure of its own.
3. Quorum: which custodians must participate, and how they verify each other. A phone call to a number from the directory, not to a number in the message that requested the opening.
4. Execution: the exact steps to reconstitute the material, referencing the tool in the envelope.
5. Record: what is written down, where, and by whom - at minimum who opened it, when, why, and which material was exposed.
6. Rotation: the deadline for rotating everything the opening exposed, and who owns that work. Opening the escrow is a compromise event by definition, because the material is now in more heads and on more disks than the design allows.
7. Re-escrow: the new material goes back into a new envelope, and the old one is destroyed.

Proving the escrow, because an unopened envelope is an assumption

This is the part that distinguishes a real control from a documented one. The escrow must be exercised, and the exercise has to prove three separate things.

1. The material is retrievable

Can the quorum actually be assembled, and does the mechanism still work? This is a people test as much as a technical one, and it is the one that surfaces the custodian who left the company in March.

2. The vault password decrypts the current repository

cd /srv/ansible

# Enumerate every vault-encrypted file in the repository.
grep -rl '^\$ANSIBLE_VAULT' --include='*.yml' --include='*.yaml' . > /tmp/encrypted-files.txt

# Prove each one decrypts. Any failure names the file.
while read -r f; do
if ansible-vault view "$f" > /dev/null 2>&1; then
  echo "OK   $f"
else
  echo "FAIL $f"
fi
done < /tmp/encrypted-files.txt

A sample is not enough. A repository with three vault IDs can have one escrowed password that works perfectly and two files it has never been able to open, and nothing about a spot check would reveal that.

The encrypt_string case needs separate attention: inline !vault values live inside otherwise-plaintext YAML, so the grep above will not match those files. Add a search for !vault and check those files by rendering them in a play with --check, or by decrypting the individual values.

3. The SSH key still authenticates

# Fingerprint of the escrowed key, to compare against what the
# managed nodes' authorized_keys actually contains.
ssh-keygen -lf /tmp/escrow/automation_key.pub

# End-to-end proof against a small, low-value group.
ansible canary -i inventory/ -m ansible.builtin.ping   --private-key /tmp/escrow/automation_key

Knowledge check