Rotating a key across a fleet without locking yourself out

Key rotation is the operation everyone agrees should happen and almost nobody performs, and the reason is not laziness. It is that the naive version of it is genuinely dangerous: a play that replaces the key on every host can, if anything goes wrong in the middle, leave you unable to reach the hosts you would use to fix it.

The safe version is not complicated. It is three phases with a gate between the second and third, and the discipline is entirely in refusing to skip the gate.

The ordering

Phase one: distribute. Add the new public key to every host. The old key remains in place and working. Nothing is removed. At the end of this phase every host should accept both keys.

Phase two: verify. Confirm, on every host individually, that the new key works — using the new key, not by observing that the run succeeded with the old one. This is the gate.

Phase three: remove. Delete the old public key from every host, now using the new key to connect. If phase two passed on every host, this cannot lock you out.

The overlap window between phase one and phase three is the safety property. During it, both credentials work, and any failure is recoverable because the old path is still there.

Phase one, in practice

- name: Phase one - distribute the new automation key
  hosts: all
  gather_facts: false
  become: true
  tasks:
    - name: Add the new automation public key
      ansible.posix.authorized_key:
        user: ansible
        key: "{{ lookup('file', 'keys/automation_2026.pub') }}"
        state: present
        exclusive: false
        key_options: 'restrict,pty,from="192.0.2.10"'

exclusive: false is doing critical work and is the default; state it anyway. exclusive: true would replace the entire authorized_keys file with only the keys in this task — which is phase one and phase three collapsed into a single irreversible step, executed against every host at once. It is the most dangerous option in this module and it is one word away from the safe one.

Phase two: verify with the new key, not with the old one

The verification has to use the credential being introduced. Anything else proves the old key still works, which was never in doubt.

$ ansible all -m ansible.builtin.ping --one-line \
--private-key keys/automation_2026 \
--ssh-extra-args '-o IdentitiesOnly=yes -o IdentityAgent=none'

web-a1.example.com | SUCCESS => {"changed": false, "ping": "pong"}
web-a2.example.com | SUCCESS => {"changed": false, "ping": "pong"}
db01.example.com | SUCCESS => {"changed": false, "ping": "pong"}
edge-07.example.com | UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Permission denied (publickey).", "unreachable": true}

IdentitiesOnly=yes and IdentityAgent=none are the part that makes this a real test. Without them, your agent may hold the old key, SSH will offer it, the host will accept it, and the run will succeed while proving nothing at all about the new key. This is the single most common way a rotation verification produces a false pass.

edge-07 above is the interesting line: reachable, authenticating with the old key, rejecting the new one. Phase one did not take there. If phase three ran against the whole fleet now, that host would be orphaned.

The hosts that were unreachable

Every real rotation has them, and they are the reason rotations stall for months.

The failure mode is procedural: the team decides phase three cannot start until every host has the new key, four hosts stay down, and the rotation sits half-complete indefinitely — with both keys valid everywhere, which is the state you were trying to leave.

The workable approach is to make the unreachable set explicit and bounded rather than treating it as a blocker.

Split the fleet by verification status. Phase three runs against the verified set, using --limit with a file generated from phase two’s output. Part XXX covers limit files; this is one of their best uses, because the list is derived from evidence rather than from a group somebody maintains.

Give the remainder an owner and a deadline. Each unreachable host becomes a ticket, not a line in a spreadsheet. A host that cannot be reached for a month is a host with a problem independent of your rotation.

Handle re-entry. A host that comes back must receive the new key before anything removes the old one there. If it returns after phase three has run elsewhere, it still has only the old key — which still works on that host, because phase three did not touch it. Re-run phase one and two against it, then phase three.

Decide the deadline in advance. At some point a host that has not been reachable for long enough is rebuilt rather than repaired. Deciding that in advance is what stops the rotation being open-ended.

Break-glass access is the precondition

Do not begin a rotation without a way into the hosts that does not depend on the key you are rotating. This is not advice about being careful; it is the condition that makes the operation recoverable at all.

Break-glass access means a path that is independent of the credential under change:

• Console or out-of-band management. IPMI, iDRAC, iLO, a hypervisor console, a cloud provider’s serial console. Independent by construction, which is what makes it the strongest option.
• A separate emergency key, held offline, restricted with from= to a jump host, and — importantly — not rotated in the same operation.
• A separate account with its own credential, present on every host for exactly this purpose.

Two properties make it break-glass rather than just a second key:

It is verified before you need it. An emergency key nobody has tested is a belief, not a control. Test it as part of preparing the rotation, on a sample of hosts, in the same way phase two tests the new key.

Its use is noticed. If break-glass access can be used without anybody knowing, it is simply a second credential and it doubles your exposure rather than protecting you. Alert on its use.

When it is a compromise, not a rotation

The three-phase ordering optimises for not locking yourself out. That is the right optimisation for a planned rotation and the wrong one for a compromised key, where the old credential is actively dangerous and every hour it remains valid is exposure.

The ordering changes:

Contain first, on the highest-value hosts. Remove the old key from production before distributing anything anywhere, accepting that automation against those hosts is broken until the new key is distributed. This is a deliberate outage of your automation, chosen over continued exposure.

Use break-glass access as the primary path, not the fallback. This is what it was established for.

Assume the attacker also has what the key could read. A compromised automation key means the fleet must be treated as compromised, which is lesson 1’s sentence arriving in practice. Rotating the key is the first step of the response, not the response.

Deciding which situation you are in is a judgement call and it should be made explicitly, by a named person, and written down — because the two procedures have opposite failure modes and running the wrong one is expensive in both directions.

Knowledge check