Secrets and runtime variables — the safe-handling pattern

An Ansible delivery pipeline has to handle four kinds of variables, and the four must live in four different places. Defaults live in the repository. Encrypted secrets live in the repository under Ansible Vault. Runtime overrides live in the environment. The vault password itself lives in the secrets store. Conflating any two of these is the mistake that puts production credentials in PR diffs, in CI logs, or in Molecule scenarios where they have no business being.

The four layers

The four layers are stacked in a strict precedence order, and each layer has exactly one place it should live:

flowchart LR
    A["role defaults"] --> B["inventory vars"]
    B --> C["vault-encrypted vars"]
    C --> D["runtime env vars"]
    D --> E["command-line extra-vars"]

• Role defaults - committed to the repository in defaults/main.yml. These are the values the role uses when nothing else overrides them. They are public: anyone with read access to the repository can see them, and that is by design.
• Inventory vars - committed to the repository in inventories/<env>/group_vars/ and inventories/<env>/host_vars/. These are the per-environment overrides: which package version, which config path, which user owns the service. They are public, version-controlled, and differ between dev, staging, and production.
• Vault-encrypted vars - committed to the repository inside an Ansible Vault file (a YAML file encrypted with ansible-vault encrypt). These are the secrets: database passwords, API tokens, TLS private keys. They are encrypted at rest, decrypted at apply time, and never written to disk in plaintext.
• Runtime env vars - passed to the apply job from the secrets store. These are the values that vary per apply: the vault password itself, ephemeral credentials minted for the apply, environment-specific endpoints that are not safe to commit.

The discipline is that each layer exists in exactly one place. Defaults do not live in environment variables. Vault variables do not live in plaintext files. Runtime variables do not live in the repository. The pipeline enforces this layering through the order in which variables resolve and through the secrets store’s protected-environment rules.

How Ansible Vault fits in the pipeline

Ansible Vault encrypts a YAML file with a password. The encrypted file is committed to the repository; the password is stored in the secrets store. At apply time, the apply job retrieves the password from the secrets store and passes it to ansible-playbook:

ansible-vault view inventories/production/group_vars/all/vault.yml

The view subcommand decrypts and prints to stdout. Inside an apply, the password is passed with --vault-password-file pointing at a file the secrets store has mounted into the runner:

ansible-playbook -i inventories/production --vault-password-file "$ANSIBLE_VAULT_PASSWORD_FILE" playbook.yml

The variable $ANSIBLE_VAULT_PASSWORD_FILE is supplied by the CI runner from the protected-environment secrets store. The file itself is mounted read-only, exists only for the duration of the job, and is never written to a log.

Three properties this enforces:

• The vault file is in the repository. The apply that production runs is replayable from the commit and the vault file alone; the password is the only out-of-band input.
• The password is in the secrets store. No developer needs to know the production vault password to read the playbook; they only need it to read the secrets inside the vault, and the lint and Molecule stages do not need it.
• The password is never echoed. The runner mounts the file as a path, not as a value; nothing in the job’s environment list or log captures it.

Runtime variables and the secrets store

Some variables do not belong in the repository at all. A short-lived AWS access key minted by OIDC federation for the apply job, a one-time database bootstrap token, an ephemeral TLS certificate issued for the apply - these are values that change per apply and have no business in version control.

The safe pattern is to pass them as environment variables to the apply job, scoped to that job only:

ANSIBLE_AWS_ACCESS_KEY_ID="$AWS_ACCESS_KEY_ID" \
ANSIBLE_AWS_SECRET_ACCESS_KEY="$AWS_SECRET_ACCESS_KEY" \
ansible-playbook -i inventories/production playbook.yml

The values come from the secrets store and exist in the runner environment for the duration of the job. They are not echoed, not logged, not written to disk, and not committed. The playbook reads them through Ansible’s environment-variable lookup (lookup('env', 'ANSIBLE_AWS_ACCESS_KEY_ID')) without ever having them appear in a vault file or a repository.

Molecule and stub credentials

The Molecule scenario must not use real production credentials, even if those credentials are in the secrets store. Molecule runs against ephemeral containers; the credentials it uses should be stub values that exercise the same code path without exposing real secrets:

provisioner:
  name: ansible
  inventory:
    group_vars:
      all:
        vault_db_password: stub-not-a-real-password
        vault_api_token: stub-not-a-real-token

The stub values satisfy the role’s lookup('env', 'VAULT_DB_PASSWORD') calls in test mode without requiring a real vault password. The verify playbook asserts that the role used the stub values; it does not assert that the values are real.

A Molecule scenario that uses real production credentials is a scenario that has escalated the credential boundary: a compromised Molecule runner now has the credentials it needs to apply against production. The discipline is that Molecule never sees real secrets, even when those secrets are technically available in the secrets store.

What the safe-handling pattern prevents

Three classes of incident the pattern prevents:

• Secrets in PR diffs. A developer who pastes a real database password into defaults/main.yml to test locally and commits it. Secret scanning catches this on push; the four-layer pattern prevents it because real secrets do not belong in defaults.
• Secrets in CI logs. An apply job that echoes $VAULT_DB_PASSWORD for debugging and the value appears in the job log. The secrets-store mounting pattern prevents it because the password is a file path, not a value.
• Secrets in Molecule scenarios. A contributor who copy-pastes a real credential into a Molecule scenario to make the verify step pass. The stub-values discipline prevents it because the scenario is configured to use stubs and the verify step asserts on those.

Production discipline

1. Role defaults are public. A default value that should be secret is a value that belongs in the vault, not in defaults.
2. Vault files are committed; vault passwords are not. The apply is replayable from the commit and the vault file alone.
3. The vault password is mounted as a file, not echoed as a value. A vault password that appears as a variable in a CI log is a vault password that needs rotation.
4. Molecule scenarios use stub credentials. A scenario that uses real secrets has escalated the credential boundary by accident.
5. Runtime variables are scoped to the job. A runtime variable that lives in the runner image is a variable that leaks into every job that uses that image.

Cross-course references

• Ansible for Production Sysadmins - Part XXI (Vault) covers the per-environment vault pattern this Part layers into the pipeline.
• Ansible for Production Sysadmins - Part XX (Inventory) covers the group_vars layout the vault files plug into.
• This course, Part XLII (Secrets) - lessons git-cicd-gitops-xlii-01 through git-cicd-gitops-xlii-06 cover the broader CI secrets-handling pattern this Part instantiates for Ansible.
• This course, Part XCIII (CredentialRotation) - lessons git-cicd-gitops-xciii-01 through git-cicd-gitops-xciii-06 cover the rotation cadence that keeps vault passwords and runtime credentials current.

Quiz