Keys and agents for unattended runs · Ansible

An automation SSH key is not a personal SSH key that happens to be used by a script. It has different properties and needs different handling, and conflating the two produces the arrangement most estates actually have: one key, on one laptop or one controller, that opens every machine in the company, with no passphrase, because a passphrase broke the cron job.

This lesson is about doing better than that without pretending the constraint that produced it does not exist. The constraint is real. An unattended run cannot type a passphrase.

Which key, for which hosts

Start with scope, because scope is the security control and everything else is mechanism.

The plugin exposes the key path as private_key_file, reachable from four directions:

Read-only / Safefour entry points, one setting— From ansible-doc -t connection ansible.builtin.ssh on ansible-core 2.21.3.

$ ansible-doc -t connection ansible.builtin.ssh

   private_key_file  Path to private key file to use for
                   authentication.
      set_via:
        cli:
        - name: private_key_file
          option: --private-key
        env:
        - name: ANSIBLE_PRIVATE_KEY_FILE
        ini:
        - key: private_key_file
          section: defaults
        vars:
        - name: ansible_private_key_file
        - name: ansible_ssh_private_key_file
      default: null

Only the last of those varies per host. That is the one to use.

Read-only / Safea key per environment, scoped in inventory

# inventory/production/group_vars/all.yml
ansible_ssh_private_key_file: ~/.ssh/automation-production
ansible_user: svc-ansible

# inventory/staging/group_vars/all.yml
ansible_ssh_private_key_file: ~/.ssh/automation-staging
ansible_user: svc-ansible

The difference between that and a single private_key_file in ansible.cfg is not tidiness. It is that the staging key is not accepted by production hosts, so a run pointed at the wrong inventory fails at the connection rather than succeeding against the wrong fleet.

That is a blast-radius control implemented in authorized_keys, and it is stronger than any control implemented in your playbook, because it does not depend on your playbook being correct. A typo in a --limit pattern, a hosts: all that inherited a merged inventory, a stale ANSIBLE_INVENTORY in a CI job — none of those can cross an environment boundary if the key cannot.

Separate the key by what it may reach, not by who is using it

Environment is the boundary that matters most, because environment is the boundary a mistake crosses. Two more are usually worth drawing:

Tier. Domain controllers, database primaries and payment hosts do not need to be reachable by the key that patches web servers. If your estate has a set of machines where an accidental run would be an incident rather than an inconvenience, they deserve a key of their own — and, ideally, a controller of their own.

Purpose. A read-only inventory-collection key that maps to a heavily restricted authorized_keys entry is a genuinely different credential from the key that runs configuration changes. command= and restrict in authorized_keys make the restriction real rather than aspirational.

What is not a useful boundary is one key per engineer for shared automation. That multiplies the number of keys production trusts without reducing what any one of them can do, and it makes rotation proportionally harder. Individual accountability belongs to the system that launches the run, not to the key at the far end.

Why the plugin recommends an agent

The connection plugin’s own description says why, and it is a limitation rather than a preference:

Read-only / Saferead the first paragraph of the plugin doc— ansible-core 2.21.3. This is the plugin describing its own constraint.

$ ansible-doc -t connection ansible.builtin.ssh

  This connection plugin allows Ansible to communicate to the target
machines through normal SSH command line.
Ansible does not expose a channel to allow communication between the
user and the SSH process to accept a password manually to decrypt an
SSH key when using this connection plugin (which is the default).
The use of 'ssh-agent' is highly recommended.

Read that carefully. It is not saying an agent is convenient. It is saying Ansible has no channel through which a passphrase prompt could be answered. The ssh process is spawned with its input wired to Ansible’s module transport, not to your terminal. There is nowhere for the prompt to appear and nowhere for your answer to go.

So with a passphrase-protected key and no agent, the run does not prompt you. It fails, and — because a failed authentication is an SSH error — it fails as UNREACHABLE, which reads exactly like a network problem and sends people to look at firewalls.

An agent resolves this by holding the decrypted key in a process, so the passphrase is entered once, by a human, and the SSH clients that follow ask the agent instead of asking you.

The unattended problem, stated honestly

An agent works beautifully for an engineer at a keyboard. Now put the same key behind a scheduler at 03:00.

There is no human to enter the passphrase. The traditional resolutions were all bad in the same direction:

What people do	What it actually means
Remove the passphrase	The key is plaintext on disk. Anyone who reads the file has the fleet.
Store the passphrase in a file next to the key	The passphrase is plaintext on disk, beside the thing it protects.
Run a long-lived agent as the service account	Any process running as that account can use the key, indefinitely, without the passphrase.
Pipe the passphrase in with an expect script	Plaintext in the process table and probably in the repository.

Notice that the first and the second are the same outcome reached by different routes. This is worth naming, because the second is frequently presented as the responsible version of the first.

The third is the least bad of the four and is what most estates run. Its honest description: the protection provided by the passphrase lasts until the agent is started, and after that the key is available to anything on the controller that can reach the agent socket. That is a real reduction in exposure — the key is no longer readable as a file, and a stolen backup or a leaked repository does not contain it — but it is not what people usually think they have bought.

Agent forwarding is not the answer, and it is worse than the problem

Sooner or later somebody proposes ForwardAgent yes so that a managed node can reach onward hosts using your key.

Do not. A forwarded agent means root on the managed node can use your key for as long as your connection is open — not read it, which is the usual reassurance, but use it, which is the only thing that matters. If the managed node is compromised, so is every host your key opens, for the duration of your run.

If a managed node genuinely needs to reach a third host, give it a credential of its own scoped to that specific need, or make the controller do the reaching and hand over only the result. ProxyJump, covered later in this part, solves the routing case without forwarding anything: the authentication happens on the controller and the bastion never sees a usable credential.

What 2.21 actually gives you

Three settings, added in 2.19, change the shape of this problem. They are recent enough that most material predates them.

Read-only / SafeAnsible can run its own agent— Executed on ansible-core 2.21.3 with no configuration file present.

$ ansible-config dump | grep SSH_AGENT

SSH_AGENT(default) = none
SSH_AGENT_EXECUTABLE(default) = ssh-agent
SSH_AGENT_KEY_LIFETIME(default) = None

The description of SSH_AGENT, from ansible-config list:

Manage an SSH Agent via Ansible. A configuration of none will not interact with an agent, auto will start and destroy an agent via ssh-agent binary during the run, and a path to an SSH_AUTH_SOCK will allow interaction with a pre-existing agent.

auto is the interesting value: an agent whose lifetime is the run. When the run ends the agent is destroyed, and with it the decrypted key. There is no long-lived agent socket sitting on the controller between runs for something else to use.

That pairs with two more:

Read-only / Safekey material, not a key path— From ansible-doc -t connection ansible.builtin.ssh on 2.21.3. Note the dependency and the exclusion.

$ ansible-doc -t connection ansible.builtin.ssh

   private_key  Private key contents in PEM format. Requires the
              'SSH_AGENT' configuration to be enabled.
      set_via:
        env:
        - name: ANSIBLE_PRIVATE_KEY
        vars:
        - name: ansible_private_key
        - name: ansible_ssh_private_key

 private_key_passphrase  Private key passphrase, dependent on
                         'private_key'.
                         This does NOT have any effect when used
                         with 'private_key_file'.
      set_via:
        env:
        - name: ANSIBLE_PRIVATE_KEY_PASSPHRASE
        vars:
        - name: ansible_private_key_passphrase
        - name: ansible_ssh_private_key_passphrase

Two constraints, both easy to trip over:

private_key takes the contents of the key and requires SSH_AGENT to be something other than none. Without an agent there is nowhere to put it.
private_key_passphrase applies to private_key only. The doc is emphatic — “This does NOT have any effect when used with private_key_file”. Setting a passphrase alongside a key path does nothing at all, silently. If your run still fails to authenticate after you set a passphrase, check which of the two options you set the key with.

Put together, an unattended run can now hold an encrypted key as a variable, decrypt it at run time, load it into an agent that exists only for the duration of the run, and never write plaintext key material to disk at any point.

Read-only / Safethe shape it takes, with the secret held by Vault

# ansible.cfg
[connection]
ssh_agent = auto
ssh_agent_key_lifetime = 300

# inventory/production/group_vars/all.yml
# Both values are vault-encrypted; see the secrets part of this course.
ansible_ssh_private_key: "{{ vault_automation_private_key }}"
ansible_ssh_private_key_passphrase: "{{ vault_automation_key_passphrase }}"
ansible_user: svc-ansible

ssh_agent_key_lifetime bounds it further — “define a lifetime, in seconds, that the key may remain in the agent” — so a run that hangs does not leave a usable key in an agent for the rest of the night.

This moves the problem; it does not delete it

Be precise about what has improved, because this pattern is easy to oversell and the next lesson in the secrets part is built on stating these boundaries honestly.

Better: no plaintext key file on the controller. No agent between runs. A stolen controller backup does not yield a usable key. The key material is encrypted at rest in the repository like any other secret, and it is revocable by removing one line from authorized_keys.

Unchanged: during the run the key is decrypted, in controller memory, and in an agent. Anything running as the automation account at that moment can use it. Whatever unlocks the vault is now the credential that protects the fleet, so it needs at least the protection the key needed — and if the vault password sits in a file beside the repository, you have renamed the problem rather than solved it.

The gain is real and it is about exposure over time and blast radius of a leaked repository, not about making the key unusable by an attacker who is already on the controller during a run. Nothing makes that true.

Under the hood

Why can private_key not work without an agent, when private_key_file can?

Because of what Ansible is allowed to hand to ssh. The client takes a key path on the command line, via -i, and there is no option that takes key material. Writing the contents to a temporary file would put plaintext key material on the controller filesystem — exactly the outcome the feature exists to avoid — and would leave it there if the run were killed.

An agent is the only interface that accepts key material through a channel that is not the filesystem and not a command-line argument. So SSH_AGENT is not a convenience wrapper around this feature; it is the mechanism that makes it expressible at all, which is why the documentation states the dependency as a requirement rather than a recommendation.

The same reasoning explains the private_key_passphrase exclusion. Decrypting a key file would mean producing a decrypted key, which would need to go somewhere. Only the agent path has somewhere for it to go.

What happens to hosts that were unreachable during a key rotation?

This is the question that turns a rotation from a task into a runbook, and it is the one people discover the hard way.

A rotation is two changes in a specific order: add the new public key to every host, then remove the old one. Between them, both keys work, which is what makes the change safe.

Now suppose eleven hosts were unreachable during step one — down for maintenance, in a datacentre with a network event, or simply switched off. They never received the new key. If you proceed to step two on the hosts you could reach, you have created a set of machines that trust only a key you are about to stop using.

They will not fail now. They will fail the next time somebody automates against them, which may be weeks later, in an incident, when that automation is the thing you were relying on to fix the incident.

The rule that follows: the removal step is gated on the addition step having succeeded everywhere, not on it having been attempted everywhere. Concretely:

Add the new key. Record the exact set of hosts that reported unreachable.
Prove the new key works — a connection-requiring task run with only the new key offered. Note that a play built from debug proves nothing here, because debug runs on the controller and never connects.
Do not remove the old key until the unreachable set is empty. Chase those hosts individually; they are now a known, bounded work item rather than an unknown future failure.
Remove the old key, then verify again.

A rotation that skips step 3 does not fail loudly. It leaves a small number of machines that will be unreachable at the worst possible moment, and no record of which ones they are.

What to check before you trust a controller

ansible-config dump --only-changed -t all | grep -i 'private_key\|ssh_agent'
grep -rn 'ansible_ssh_private_key_file\|ansible_private_key' inventory/
ls -l ~/.ssh/                       # file modes on any key present
ssh-add -l                          # what a long-lived agent is currently holding

ssh-add -l on a shared controller is frequently the most alarming of the four. An agent holding a key with no lifetime, in an account several people can reach, is a fleet credential available to all of them without anybody having been granted it.

Knowledge check

Why this matters

The single flat automation key is the most common serious finding in an Ansible estate, and it is never the result of a decision. It is the result of a series of individually reasonable steps: one key because one team, no passphrase because cron, one agent because convenience, no rotation because nobody knows what would break.

Each step is defensible in isolation and the destination is a credential that opens every machine in the company, held in plaintext, with no expiry and no record of who can read it.

The controls in this lesson are cheap in comparison, and two of them cost nothing at all: scoping the key per environment in group_vars is a five-line change that turns “the wrong inventory ran against production” from an incident into a failed connection, and knowing that ssh_agent = auto exists means the passphrase argument no longer has to be lost.