When the secret should not be in the repository at all · Ansible

Every lesson in this part so far has worked on the same premise: the secret is in your repository, encrypted, and the job is to keep it that way and to stop it escaping once decrypted.

This lesson questions the premise. A secret that is never in the repository cannot leak from the repository, cannot be committed by accident, cannot sit in a clone on a laptop that gets stolen, and does not need a password you have to distribute.

The property that makes this work

Lookups run on the controller, not on the managed node. That is the whole architecture in one sentence, and it is worth proving rather than asserting.

Read-only / Safea lookup resolves for a host that is never contacted— Executed on ansible-core 2.21.3. The play targets 192.0.2.10, documentation space, unreachable by design. The lookup value is the controller user.

$ ansible-playbook -i inv.ini lk.yml

TASK [Show the lookup value on the controller before any connection] ****
  "msg": "lookup resolved to: ebrandi"

TASK [Now actually connect] ********************************************
fatal: [web1.example.com]: UNREACHABLE! => {"changed": false,
"msg": "Task failed: Failed to connect to the host via ssh:
ssh: connect to host 192.0.2.10 port 22: Connection timed out",
"unreachable": true}

The lookup produced a value; the host was never reachable. Whatever the lookup did, it did on the controller.

Three consequences follow, and they are the reason this pattern is practical at all:

Managed nodes need no access to the secret store. They do not need network reachability to it, credentials for it, or any client software. Four hundred hosts do not become four hundred secret store clients.

One credential fetches for the whole fleet. The controller authenticates once and distributes what it retrieves. That is one identity to manage, audit and revoke rather than one per host.

The controller becomes even more sensitive. It now holds the identity that can retrieve production credentials on demand. The previous lesson’s conclusion applies with more force: a controller that can decrypt or retrieve production secrets is a production system.

Retrieving a secret

The community.hashi_vault collection is the reference implementation. Every cloud provider ships an equivalent collection, and the shape is the same in each.

Read-only / Safethe KV v2 lookup, with the return structure the docs describe

- name: Deploy with a credential fetched at run time
hosts: appservers
vars:
  # Runs on the controller. Nothing is stored in this repository.
  db_creds: "{{ lookup('community.hashi_vault.vault_kv2_get',
                       'billing/database',
                       engine_mount_point='secret',
                       url='https://vault.example.com:8200',
                       auth_method='token',
                       token=lookup('env', 'VAULT_TOKEN')) }}"
tasks:
  - name: Write the application credentials
    ansible.builtin.template:
      src: credentials.j2
      dest: /etc/billing/credentials
      owner: billing
      mode: '0600'
    vars:
      db_password: "{{ db_creds.secret.password }}"
    no_log: true

Two details from the plugin documentation worth getting right first time.

The return is a dictionary, not a string. The documented accessible keys are secret — “The data field within the data field. Equivalent to raw.data.data” — plus data, metadata and raw. So the value you want is db_creds.secret.<key>, and a lookup returning something that looks nothing like a password is almost always this.

engine_mount_point defaults to secret. If your KV engine is mounted elsewhere, every path is wrong in a way that presents as “the secret does not exist”.

The collection provides a family of lookups — hashi_vault, vault_kv1_get, vault_kv2_get, vault_read, vault_list, vault_login, vault_write, vault_token_create and vault_ansible_settings — plus modules for managing the store itself, including dynamic database credentials. Read the one you need; do not extrapolate parameter names from a sibling.

A lookup in a play-level vars block runs once per host, every play

vars at play level is evaluated lazily, per host, when something references the variable. A play over 400 hosts referencing db_creds in a task can therefore hit the secret store 400 times for one value.

Usually harmless, occasionally not: rate limits, audit logs filling with identical entries, and — if the path issues dynamic credentials — 400 distinct credentials created where you wanted one.

That last case is the one to watch, because it is silent and it is a leak of a different kind: leases you did not intend to create, expiring on their own schedule, with nothing tracking them.

Fetch once when the value is shared:

  tasks:
    - name: Fetch the credential once for the whole play
      ansible.builtin.set_fact:
        db_creds: "{{ lookup('community.hashi_vault.vault_kv2_get',
                             'billing/database') }}"
      run_once: true
      delegate_to: localhost
      no_log: true

run_once with delegate_to: localhost states the intent plainly: this happens once, on the controller. Note that the fact caching caveat from the previous lesson applies — if fact caching is enabled, this value can be written to the cache backend in plaintext.

Static secrets versus dynamic credentials

Fetching a stored password at run time is an improvement. It is not the main prize.

The larger change is a credential that did not exist before the run and expires after it. The store creates a database user on demand with a lease of, say, one hour, hands it over, and revokes it when the lease ends.

	Encrypted file in the repository	Fetched static secret	Dynamic credential
In the repository	yes, encrypted	no	no
Lifetime	until someone rotates it	until someone rotates it	minutes to hours
Revocation	rotate and redeploy everywhere	rotate at the store	expires on its own
A leaked run log	exposes a live credential	exposes a live credential	exposes an expired one
Attribution	none	which identity fetched it	which identity, and this exact lease
Works when the store is down	yes	no	no

The fourth row is the one that reframes the whole part. Every leak route in this part’s first lesson — the log file, the CI artifact, the pasted terminal, the --diff output — leaks a credential that has already expired if the credential lived for an hour. The exposure does not disappear, but its value decays to nothing without anybody doing anything.

Note also that dynamic credentials do not remove the need for no_log, Vault or output discipline. They shorten the window during which a leak matters. That is a different kind of protection from the others in this part, and it composes with them rather than replacing them.

Under the hood

How does the controller authenticate to the store without a stored credential — which would just move the problem?

The good answers are all forms of ambient identity: something the platform already gives the machine, which the store can verify independently.

Cloud instance identity. The controller proves it is a particular instance using metadata the cloud provider signs. Nothing is stored.
Kubernetes service account. A projected token the API server issues and the store validates against it.
AppRole with a response-wrapped secret ID. The provisioning system delivers a single-use wrapped token; unwrapping it twice fails, so interception is detectable rather than silent.
TLS client certificate. The machine’s own certificate, issued by your internal CA, with the rotation problem handled by whatever manages that CA.

The property shared by all four: the credential is bound to the machine and cannot be copied off it usefully. That is the qualitative difference from a password file, which is a string and travels like one.

The bad answer is a long-lived token in an environment variable or a file on the controller, which is a password file with extra steps. It is a reasonable starting point while you build something better, and it should be recorded as a known gap rather than presented as the design.

This is also where the vault_login lookup fits: authenticate by one method, obtain a short-lived token, and use that token for the rest of the run. It keeps the durable credential out of the individual lookups.

The store is now in the critical path of every run

This is the trade-off, and it is stated last in the curriculum for this part because it is the one that gets discovered rather than planned.

If the secret store is down, your automation does not run. Not degraded — stopped, at the point where the lookup fails, before the tasks that depend on it.

Which produces a scenario worth thinking through before it happens: the secret store is down, and the runbook for restoring it involves running automation, and that automation fetches its credentials from the secret store.

Four things to establish before you depend on this:

What is the store’s availability target, and is it higher than your automation’s? If the store is a single instance somebody set up for a proof of concept, your fleet automation now inherits that.
Is there a break-glass path? A sealed, offline copy of the credentials needed to recover the store itself, held under a process that makes using it visible — two-person access, an alert on retrieval. Not a file on the controller.
Does recovering the store depend on the store? Diagram it honestly. Circular dependencies in recovery paths are invisible until the day they matter, and then they are the whole incident.
What happens to in-flight runs? A run that fetched a credential at the start and lost the store mid-run may continue happily or may fail on the next lookup, depending on where the lookups are. Know which.

The honest summary: you have traded a durable local risk — a secret at rest that could leak — for an availability dependency. For most estates that is the right trade. It is not a free one, and pretending otherwise is how recovery runbooks end up circular.

Choosing, per secret, in one pass

Not everything needs the same treatment, and trying to move all of it at once is how these projects stall. Ask three questions per secret:

Does anything support dynamic issuance for it? Databases, cloud APIs and internal services increasingly do. If yes, that is the strongest option and worth the migration effort on its own.

How often does it change, and how many places consume it? Something rotated monthly and consumed by twenty playbooks is painful as an encrypted file and easy as a lookup. Something set once at build time and consumed by one role is fine in Vault.

Would a failure to fetch it be worse than the exposure it avoids? A credential needed by the disaster recovery playbook is exactly the one you do not want behind a network dependency.

A realistic destination for most estates is a mix rather than a migration: dynamic credentials where the target supports them, fetched secrets for frequently rotated shared material, and Vault for bootstrap and break-glass material that must work when nothing else does.

Whatever the mix, keep the manifest from the vault operations lesson current — which secret lives where, who can retrieve it, and how it is rotated. That document is what makes the mix maintainable rather than archaeological.

What happens if the secret store is unreachable during a fleet run?

The lookup fails and the play stops at that point, which is the correct behaviour and still leaves you with decisions.

If the lookup is in play-level vars, it resolves at first use, so the failure lands wherever that is — potentially after several tasks have already run against every host. The assertion pattern from the encrypt_string lesson applies directly: put a pre_tasks check that the credential was retrieved, so the run refuses to start rather than part-configuring the fleet.

If it is a set_fact with run_once, the failure is at that task, near the top, and the blast radius is whatever ran before it.

Either way the immediate question is the one from the SSH part: what state is the fleet in? Some hosts changed, some not, and the run stopped somewhere in between.

Do not respond by pasting a credential in as an extra variable to get the run finished. That is the fastest available fix and it takes a well-designed system back to the worst pattern in this part — a secret on a command line, in shell history, in the process table, and quite possibly in a chat message where somebody shared the command that worked.

The prepared answer is the break-glass path from the callout above: retrieved deliberately, used once, with the retrieval visible to somebody other than the person retrieving it, and the credential rotated afterwards because it has now been handled outside the normal channel.

If you do not have that path, the honest response to the incident is to wait for the store, and to write the break-glass path afterwards while the memory of waiting is fresh.

Knowledge check

Why this matters

This part opened by enumerating where secrets go and spent six lessons on controls that each cover one hop. This lesson removes several hops at once by declining to put the secret in the repository at all.

That is genuinely better, and it is not free, and the cost lands somewhere teams do not usually look. You have converted a confidentiality problem into an availability problem: instead of a secret at rest that might leak, you have a network dependency that might be down, in the critical path of every run including the ones you make during an incident.

For most estates that is the right trade, especially where dynamic credentials are available and the exposure window collapses from indefinite to an hour. What makes it a decision rather than an accident is diagramming the recovery path before you depend on it, and being able to answer plainly what happens to your fleet automation on the day the store is the thing that is broken.