Runbook: Rotate the fleet automation SSH credentials

The automation SSH key is the credential that lets one host change every other host. Rotating it is routine and it is also the change most likely to lock your automation out of your own estate, because the tool you would use to fix the problem is the tool the change just broke.

The whole procedure is built around one rule: add everywhere, verify everywhere, then remove. Every failure mode in this runbook comes from compressing those three phases into one.

When to use this runbook

• Scheduled rotation on the estate’s key lifetime.
• An operator with controller access has left the team.
• The key is suspected or known to be compromised (see the ordering note below - this case is different).
• Migrating to a stronger key type.
• Replacing a controller that generated its own key.

Ordering when the key is compromised

If this rotation is a response to compromise, the phases stay in the same order but the clock changes. Adding the new key is still first, because you need a working path. But removal is no longer a scheduled tail-end task - it is the point of the exercise, and a host that cannot be reached to have the old key removed is an actively exposed host, not a paperwork item. Escalate it the same hour, do not carry it in the exception list for a week.

Blast radius

Every host that trusts the automation key. That is usually more than the primary inventory contains: decommissioned-but-alive machines, appliances, a jump host, a partner-managed box someone added once.

Step 2 exists to find those. Rotating only what is in inventory leaves the old key valid somewhere, which means the rotation did not happen.

Inputs

• The current automation private key path and its fingerprint.
• The complete list of hosts that trust it.
• The break-glass access path, tested.
• The change window, and a second window for removal.

Step 1: Generate the new keypair

sudo -iu ansible
ssh-keygen -t ed25519 -f /home/ansible/.ssh/id_ed25519_new \
-C "ansible-controller rotation 2026-08 - replaces SHA256:REPLACE_ME" -N ''

ssh-keygen -lf /home/ansible/.ssh/id_ed25519_new.pub
ssh-keygen -lf /home/ansible/.ssh/id_ed25519.pub

Record both fingerprints. You will compare against them at every later step, and “the old one” stops being an unambiguous phrase about twenty minutes into a rotation.

The comment field is not decoration. It is what tells the next person looking at an authorized_keys file which key this is and what it replaced.

A passphrase-less key is the norm for unattended automation; the protection is file permissions and the fact that the controller is itself a controlled host. If your estate uses an agent with a passphrase, load it now and confirm ssh-add -l lists the new key before continuing.

Step 2: Snapshot who trusts the old key

- name: Record which hosts trust the automation key
hosts: all
gather_facts: false
tasks:
  - name: Read the automation authorized_keys
    ansible.builtin.slurp:
      src: /home/ansible/.ssh/authorized_keys
    register: akf
    become: true

  - name: Report the keys present
    ansible.builtin.debug:
      msg: "{{ akf.content | b64decode | split('\n') | select('search', 'ssh-') | list | length }} key(s)"

The important output of this step is not the count. It is the list of hosts that answered, compared against the list of hosts you believed existed. Anything trusting the key that is not in inventory goes to its owner now.

Step 3: Add the new key alongside the old

- name: Stage the new automation key
hosts: all
gather_facts: false
become: true
tasks:
  - name: New public key is authorised, old one untouched
    ansible.posix.authorized_key:
      user: ansible
      key: "{{ lookup('file', '/home/ansible/.ssh/id_ed25519_new.pub') }}"
      state: present
      exclusive: false

Run it, and note the recap. Unreachable hosts here are the whole point of Step 5.

Step 4: Verify the new key, using only the new key

ansible all -m ping -o \
-e 'ansible_ssh_private_key_file=/home/ansible/.ssh/id_ed25519_new' \
-e 'ansible_ssh_extra_args="-o IdentitiesOnly=yes"'
echo "exit=$?"

IdentitiesOnly=yes is what makes this a real test. Without it, SSH offers every key the agent holds and every default identity file, so a host that does not have the new key still authenticates with the old one and reports SUCCESS. You would then remove the old key from a host that never received the new one.

Expected: SUCCESS for every host, exit code 0. Exit code 4 means at least one host was unreachable; that host does not proceed to removal.

Step 5: Write down the exceptions

Every host that failed Step 3 or Step 4 goes on a named list with an owner. Not a mental note, not a scrollback buffer.

ansible all -m ping -o \
-e 'ansible_ssh_private_key_file=/home/ansible/.ssh/id_ed25519_new' \
-e 'ansible_ssh_extra_args="-o IdentitiesOnly=yes"' \
2>&1 | grep -E 'UNREACHABLE|FAILED' | awk '{print $1}' | sort -u \
> rotation-exceptions.txt
wc -l rotation-exceptions.txt

These hosts keep the old key. They must therefore be excluded from Step 8 by name, and they block the closeout in Step 11. A rotation with an open exception list is a rotation in progress, not a completed one.

Step 6: Cut the controller over

cd /home/ansible/.ssh
cp -a id_ed25519 id_ed25519.pre-rotation
cp -a id_ed25519.pub id_ed25519.pub.pre-rotation
install -m 0600 id_ed25519_new    id_ed25519
install -m 0644 id_ed25519_new.pub id_ed25519.pub
ssh-keygen -lf id_ed25519.pub

Keep the old private key on disk for now, named so nobody mistakes it for the live one. It is your rollback, and Step 10 is where it goes away.

Step 7: Full read-only pass on the new key

ansible all -m ping -o | tee rotation-new-key-proof.txt
echo "exit=$?"
grep -c SUCCESS rotation-new-key-proof.txt

This is now the default identity, so no extra flags. Compare the SUCCESS count against the host count from the inventory. They must match, minus the exception list.

Run something slightly heavier as well - a check-mode playbook run - because ping proves authentication and not much else. A key that authenticates but whose account lost sudo in the same change window will pass ping and fail everything real.

Step 8: Remove the old key, in waves

- name: Retire the old automation key
hosts: all
gather_facts: false
become: true
serial: "20%"
tasks:
  - name: Prove the CURRENT connection uses the new key before removing anything
    ansible.builtin.command: ssh-add -L
    delegate_to: localhost
    run_once: true
    changed_when: false

  - name: Old public key is no longer authorised
    ansible.posix.authorized_key:
      user: ansible
      key: "{{ lookup('file', '/home/ansible/.ssh/id_ed25519.pub.pre-rotation') }}"
      state: absent

  - name: Connection still works after removal
    ansible.builtin.ping:

# Build an explicit exclusion pattern from the exception list
LIMIT="all$(sed 's/^/:!/' rotation-exceptions.txt | tr -d '\n')"
echo "$LIMIT"

ansible-playbook retire-old-key.yml --limit "$LIMIT" --list-hosts
ansible-playbook retire-old-key.yml --limit "$LIMIT" --diff

serial: "20%" means a mistake stops after a fifth of the fleet rather than all of it. The final ping task inside the same play is the wave gate: if removing the old key breaks the connection, the play fails on that batch and the remaining batches never run.

Verified behaviour worth relying on: with serial set and a batch failure that exceeds max_fail_percentage, later batches are not attempted and the untouched hosts do not appear in the recap at all. Do not read an empty recap line as “that host was fine”.

Step 9: Prove the old key is dead

Verification that cannot fail is not verification. This one can:

for h in web01.example.com db01.example.com cache01.example.com; do
if ssh -o BatchMode=yes -o IdentitiesOnly=yes \
      -i /home/ansible/.ssh/id_ed25519.pre-rotation \
      -o ConnectTimeout=10 "ansible@$h" true 2>/dev/null; then
  echo "STILL TRUSTED: $h"
else
  echo "revoked OK:    $h"
fi
done

Sample at least one host per group. STILL TRUSTED on any of them means the removal did not take on that host - most often because the file it edited is not the file sshd reads.

ansible all -b -m command \
-a "sshd -T -C user=ansible,host=%h,addr=192.0.2.1" -o \
| grep -iE 'authorizedkeysfile|authorizedkeyscommand'

If authorizedkeyscommand is anything other than none, keys come from a helper - LDAP, a key server, a certificate authority - and editing files on the host revokes nothing. Revoke at that source instead, and treat every file-based step above as having done nothing for those hosts.

Step 10: Destroy the old private key

shred -u /home/ansible/.ssh/id_ed25519.pre-rotation
rm -f /home/ansible/.ssh/id_ed25519.pub.pre-rotation
ls -l /home/ansible/.ssh/

Then deal with copies: any controller backup taken during the window contains the old private key. Either purge those snapshots or record that they contain a retired credential and when they expire. A key “rotated” out of the live filesystem and left in a backup that anyone can restore has not been rotated.

Step 11: Close the exception list

Every host on rotation-exceptions.txt is now one of:

• Done - it came back, got the new key, and had the old one removed. Re-run Steps 3, 4 and 8 against it individually.
• Quarantined - it cannot be reached and has been formally recorded as trusting a retired key, with an owner and a date. If the rotation was due to compromise this is an open security finding, not an operations backlog item.
• Decommissioned - it is gone, and its removal is recorded.

There is no fourth category. “We will get to it” is how a retired key stays valid on a host for two years.

Rollback

cd /home/ansible/.ssh
install -m 0600 id_ed25519.pre-rotation id_ed25519
install -m 0644 id_ed25519.pub.pre-rotation id_ed25519.pub
ansible all -m ping -o; echo "exit=$?"

Common patterns

Escalation

Escalate when:

• A host is unreachable at removal time. Removing a credential from a machine you cannot verify creates an unmanageable host.
• The rotation is a compromise response. Timelines change and the security owner runs the decision, not the change queue.
• A host trusts the automation key but is not in inventory.
• Break-glass access has not been tested. Test it before Step 8, not during the incident that follows Step 8.