Secrets separated by environment · Ansible

Part XXI covers what Ansible Vault is, how to encrypt with it, how to rekey, and what the vault id labels do and do not enforce. This lesson assumes all of that and asks one structural question:

Can a developer with development credentials decrypt production secrets, even by accident?

The answer you want is no, and getting there is less about ansible-vault options than about which password file is on which machine.

The layout

Encrypted material lives inside the environment directory, beside the plaintext variables it belongs with:

inventories/
  production/
    group_vars/
      all/
        main.yml            # plaintext: ntp servers, log target
        vault.yml           # encrypted, vault id: prod
      webservers/
        20-limits.yml
        vault.yml           # encrypted, vault id: prod
  staging/
    group_vars/
      all/
        main.yml
        vault.yml           # encrypted, vault id: stage
  development/
    group_vars/
      all/
        main.yml
        vault.yml           # encrypted, vault id: dev

Three properties matter here and none of them is about encryption.

One vault id per environment, with a distinct password. Not one password with three labels. The label does not restrict anything; the password does.

A separate vault.yml rather than encrypting main.yml in place. An encrypted whole file cannot be diffed or reviewed, so putting the non-secret variables in it makes every ordinary change invisible to review. Split them and the plaintext file stays reviewable.

Encrypted files sit in the environment tree, not in a shared group_vars/ beside the playbook. A shared one applies to every environment — which means every environment needs the password to read it, which means there is no separation.

What the label actually does

This is the part people get wrong, and the consequence is a scheme that looks separated and is not.

A vault id label is recorded in the file header:

Read-only / Safethe header carries the label— Real output from ansible-core 2.21.3. The 1.2 format records the vault id as the fourth field.

$ head -1 inventories/production/group_vars/all/vault.yml

$ANSIBLE_VAULT;1.2;AES256;prod

That label tells Ansible which supplied password to try first. By default it does not restrict which passwords may be tried at all. Offer the production password under the label dev and the file opens anyway:

Read-only / Safethe label does not have to be true— Real output from ansible-core 2.21.3. The label says dev; the file behind it is the production password. The prod-labelled file decrypts.

$ ansible-vault view --vault-id dev@~/.vault/prod inventories/production/group_vars/all/vault.yml

db_password: REDACTED-prod

Part XXI’s vault-id lesson covers DEFAULT_VAULT_ID_MATCH, the setting that makes the label binding, and it is worth turning on. But do not mistake it for the control. The reason a developer cannot read production secrets is not that a label mismatched — it is that the production password was never on their machine.

Separation is a property of distribution, not of syntax

Write down, for each environment, which machines hold its vault password. The list should be short and it should not overlap where it matters:

Password	Developer laptop	CI runner (test)	CI runner (deploy)	Production controller
`dev`	yes	yes	no	no
`stage`	no	yes	yes	no
`prod`	no	no	yes, restricted	yes

The row that does the work is prod. If the ordinary test runner can decrypt production secrets, then anyone who can open a pull request that runs CI can print them — and that is a much larger group than the people you meant to trust with production credentials.

Part XXXVIII’s secret-scanning lesson deals with what happens once a secret is exposed; the point here is that this table is the thing that decides how many people could expose it.

What a wrong vault id does to a run

This is the good news, and it is worth knowing precisely, because it means the mistake fails in the safest possible way.

Group variables are decrypted while the play is being set up, before any host is contacted. A wrong password therefore stops the run before a single connection is attempted:

Configuration changeproduction inventory, development vault id— Real output from ansible-core 2.21.3. The play stops at variable load. No task ran, no host was contacted, and there is no PLAY RECAP because no play completed.

$ ansible-playbook -i inventories/production --vault-id dev@~/.vault/dev playbooks/webservers.yml

PLAY [Configure web tier] ******************************************************
[ERROR]: Decryption failed (no vault secrets were found that could decrypt).

Exit status 4. Zero tasks attempted, zero hosts touched, no recap.

That is the behaviour you want from a wrong-credential mistake, and it is worth stating in the runbook: if you see this error against production, you supplied the wrong vault id — nothing has been changed.

The same property applies to inventory inspection, which is easy to forget when you are only trying to look at something:

Read-only / Safeyou cannot even list the inventory without the password— Real output from ansible-core 2.21.3. group_vars loading is not optional, so an encrypted file in the tree makes the password mandatory for inspection too.

$ ansible-inventory -i inventories/production --graph

[ERROR]: Attempting to decrypt but no vault secrets found.

The static gates do not catch a wrong vault id

Here is the gap, measured, and it matters because it is the gate CI runs first.

ansible-playbook --syntax-check with the wrong vault id against the production inventory exits 0. Syntax checking does not decrypt group_vars, so the credential error that stops a real run does not surface at all.

The practical consequences:

A CI job whose only gate is --syntax-check will pass a pipeline whose vault configuration is broken, and the failure appears at deploy time against production.
Add ansible-inventory -i <env> --graph > /dev/null as a separate gate. It loads and decrypts group_vars, so it fails when the credential is wrong. Redirect the output — see the next callout for why.

Part XXXVIII’s pipeline lesson makes this one of the ordered stages. The principle it illustrates is the one that lesson is built on: each gate catches a class the previous one structurally cannot.

Under the hood

Where the password itself lives per environment is the question that decides whether any of this holds, and Part XXI’s controller lesson makes the full argument. The environment-specific part is worth restating.

A password file beside the repository, readable by the user who can read the repository, adds nothing: anyone who can check out the code can decrypt it. That is acceptable for dev, arguably acceptable for stage, and not acceptable for prod.

For production the useful forms, in increasing order of strength:

A file outside the repository, mode 0600, owned by a service account that operators cannot log in as. Now decryption requires being that account, not merely having the checkout.
An executable password source — --vault-id prod@/usr/local/bin/get-prod-pw — which fetches the secret from a broker at run time and can log the request. Ansible executes it and reads the password from its standard output; nothing is stored on disk.
A short-lived credential from a secret manager, so a copied password stops working. Part XXI covers the lookup-plugin route and its trade-offs, notably that the secret is then in the controller’s memory and in the play’s variable space either way.

None of these help if the same account holds every environment’s credential, which is why the distribution table earlier is the first thing to fix and the encryption details are the second.

What happens if one controller has to run every environment?

Small teams have exactly one machine that runs automation, and the neat separation above starts to look aspirational.

What you lose is real: a controller holding all three passwords can decrypt all three environments, and offering both ids on one command line is enough. Verified on 2.21.3 — a production run with --vault-id dev@... --vault-id prod@... succeeds, because Ansible tries each supplied secret in turn. The labels do not stop it.

What you can still keep:

Distinct passwords, even on one machine. Then a leaked development password is still not a production credential, which is the case that actually happens — development passwords end up in wikis, chat and scripts far more often than production ones.
Distinct file ownership. The production password file owned by a separate service account, with sudo required to read it, means the ordinary interactive session cannot decrypt production even though the machine can.
One vault id per invocation. Make the wrapper or the CI job supply exactly the id for the environment being targeted. Passing all three “so it always works” removes the last of the separation, and it removes the good failure mode from the previous section along with it.
DEFAULT_VAULT_ID_MATCH on. With one controller it stops being a nicety: it is what makes a mislabelled invocation fail instead of quietly working.

And be honest in the README about what the arrangement does not achieve, so that the next person does not inherit a separation scheme and believe it is stronger than it is.

The layout

What the label actually does

What a wrong vault id does to a run

Knowledge check