Credentials the operator never sees · Ansible

There is one thing a platform does that a controller genuinely cannot, and this lesson is about that one thing.

A person can launch a job that authenticates with a credential they are not able to read.

Every other capability in this part has a decent command-line equivalent. This one has an approximation with a hole in it, and the hole is where most estates keep their production root key.

Why a controller cannot do this

The argument is short. On a shared controller, ansible-playbook runs as you. Anything it can read, you can read.

Read-only / Safethe vault password file that lets a play decrypt also lets you decrypt— Verified on ansible-core 2.21.3 in a scratch directory with a deliberately fake secret. Same password file, different command, no additional privilege.

$ ansible-vault view --vault-password-file .vault-pass secrets.yml

---
db_admin_password: REPLACE_ME_DB_PASSWORD

The same is true of every other credential the run depends on. If your SSH agent holds the production key while the playbook runs, ssh can use it too. If the run reads a cloud token from an environment variable, so can env.

This is not a hardening failure. It is what “runs as you” means. On a controller, use implies read, and the only way around it is to run the job as an account you cannot become — which is the sudo-plus-wrapper construction from lesson 3, and which is worth building, and which fails completely against anyone with root on that box.

What the platform does instead

The mechanism has three parts, and it is worth knowing all three because the security property depends on each.

Encrypted at rest, and never returned. AWX encrypts passwords and key material in its database and never makes secret information visible via the API. A secret field, read back through the API, comes out as a placeholder rather than a value. There is no supported call that returns it.

Attached, not referenced. A credential is attached to a job template as an object. The template stores a relationship, not a path or a value. Nothing in the template needs to contain the secret in order to use it.

Injected at launch, into a process the user does not control. When a job starts, the platform decrypts the credential and injects it into the run — an SSH key written to a temporary file for the duration, a vault password supplied to the process, a cloud token placed in the environment — inside the execution environment, then removes it when the job ends.

Attaching a credential to a template requires the Use role on that credential. Use lets you employ the credential in a job template without viewing its sensitive content. That single role is the whole property:

The end user gets the benefit of the secret without the secret.

What still leaks

The property is real and it is narrower than people assume. An injected credential is protected from being read at rest. It is not protected from a playbook that prints it.

Everything from Part XXI about how secrets leak applies unchanged, and a platform makes two of the leaks worse because the output is now stored, searchable, and readable by anyone with Read on the job.

A command or shell task with the secret in its arguments. The task line appears in the output. A platform records that output permanently.

--diff on a rendered template. Diff mode prints file contents, including the credential your template just wrote into a config file. Set diff: false on tasks that render secrets, as Part XXV covers.

A registered result printed with debug. no_log: true on the task that handles the secret, and remember it applies to the task, not to the variable — a later task that touches the same value needs its own.

Verbosity. A job template has a verbosity field. At higher levels the module arguments are printed, which is exactly where the credential is.

The lifetime of an injected SSH key

Follow one machine credential through a run, because knowing where the material exists tells you what a compromise reaches.

At rest, the private key sits encrypted in the platform database. The encryption key is on the platform host. Compromise both and you have every credential; this is the concentration risk, discussed below.
At launch, the platform decrypts the key in memory and starts the execution environment.
During the run, the key is written to a temporary file inside the execution environment, or handed to an agent process started for that job. It exists on disk, briefly, in a container the requesting user has no shell in.
On the wire, nothing changes. This is ordinary SSH — the key authenticates to the managed node exactly as it would from your laptop.
On the managed node, nothing changes either. The node sees a public key it trusts. It has no idea a platform is involved.
At the end, the temporary material is removed with the execution environment.

Two conclusions follow, and both are worth stating plainly.

The managed nodes are unchanged. A platform is not a different security architecture on the fleet side. The same key, the same authorized_keys, the same become configuration. Everything Part XX and Part XIX taught still describes what happens.

The protection is process isolation, not cryptography. The user cannot read the key because they cannot get inside the container it is written into, not because of anything mathematical. It follows that anyone with a shell on the platform host, or the ability to make the platform run arbitrary code, has the credentials. A job template that runs an arbitrary playbook from a repository somebody can push to is exactly that ability.

The risk you accept

Put all the credentials in one system and that system becomes the most valuable host in the estate. This is not a reason to avoid doing it — scattering them across fourteen controllers is worse — but it is a trade you should make deliberately.

What the attacker gets. Not one environment: all of them. Not one key: the key, the vault passwords, the cloud tokens, the registry credentials. Plus the ability to run arbitrary automation against everything, with a job history telling them exactly which templates reach which hosts.

How they get it, in practice. Rarely by breaking the encryption. Usually by one of:

An account with Admin on a project, pointed at a repository they can push to. The platform runs what the repository contains, with whatever credentials the template holds.
An account with Admin on a job template, editing it to run a different playbook with the same credentials.
A shell on the platform host, which holds the encryption key.
Backups of the platform database, stored somewhere with weaker access control than the platform itself.

What mitigates it. Nothing eliminates concentration; these reduce the blast radius of a compromise:

Control	What it buys
Distinct credentials per environment, never a fleet-wide one	A compromised staging credential does not reach production
Credential Admin held by a small named group	Fewer accounts can create a template that uses the production key
Projects pinned to a protected branch or tag	Pushing code is not enough to make the platform run it
An external secret manager as the source of truth	The platform holds a short-lived lease rather than the permanent secret
Short-lived certificates instead of long-lived keys	A stolen credential expires on its own
Monitoring on credential use, not just on job success	The unusual run at 04:00 is visible as an event

The last two are where mature estates end up. Part XXI covered external secret managers; the relevant point here is that a platform integrated with one holds a reference plus the authority to fetch, rather than the secret itself — which converts “an offline copy of everything” into “an online capability we can revoke”.

What happens if you need to know which secret a job actually used?

You will need this during an incident — a managed node rejects an authentication and you need to know whether the credential is wrong, the key is stale, or the target is unexpected.

The platform answers the first part: the job record shows which credential objects were attached. That is usually enough to conclude “it used the production machine credential”, which is what you were asking.

It does not tell you the value, by design, and this is where people get stuck. There is no supported route to read the stored secret back, so the question “is the key in the credential the same one on the host” cannot be answered by looking at both. You have to compare the public halves, or test the credential against a known-good host, or replace it.

The consequence is a genuine operational discipline: a credential in a platform is only debuggable if its public half is recorded somewhere outside the platform. Store the SSH public key and its fingerprint in your documentation when you create the credential. Fingerprints are not secret and they are the only thing that makes “which key is this” a five-second question rather than a twenty-minute one.

The same applies to vault credentials, where the equivalent is recording which vault identity a credential corresponds to. A vault password with no recorded identity is a secret you can use and cannot reason about.

Knowledge check

Why this matters

Most discussions of secret management get stuck on storage: where the secret lives, how it is encrypted, which vault product to buy.

The property that changes how an estate operates is not storage. It is the separation of the person who benefits from a secret from the person who holds it. Once fifteen people can run the restore job and none of them know the database password, three durable things follow.

Departures stop being credential events, so rotation stops being a crisis, so rotation actually happens.

Access reviews become meaningful, because “who can use production credentials” and “who knows production credentials” are finally different lists and both are enumerable.

And the estate stops depending on the assumption that fifteen people are all careful with something they were shown once, on a controller, two years ago.

You pay for it with concentration: one system now worth more than any host it manages, whose backups are worth the same, and which must be defended accordingly. That is a trade worth making — but only if you make it consciously, and only if you make the same list of what an attacker gets that this lesson just made.