Rotating a secret across three hundred hosts · Ansible

The naive rotation:

Change the secret at the provider.
Run the playbook to push the new value to 300 hosts.

Between step 1 and the moment the last host reloads, every host that has not yet been updated is authenticating with a value that no longer works. With a fleet of 300 and a play that takes eleven minutes, that is an eleven-minute authentication outage, applied to a progressively smaller fraction of the fleet, starting the instant step 1 completes.

It is also not recoverable by rolling back, because the old value does not exist any more. Part XLVIII lists provider-side secret rotation among the changes with no reverse.

The problem is ordering, and it has no atomic solution

A shared secret is held in two places: the authority that validates it, and every consumer that presents it. Changing both simultaneously across 300 machines is not something Ansible can do — not because of a limitation in Ansible, but because there is no coordinated moment across 300 independent SSH sessions.

So the sequence has to be constructed so that no moment requires simultaneity.

Step	What runs	Estate state
1. Generate	The new value is created, not yet trusted	Old value works
2. Accept both	The authority accepts old and new	Both work
3. Deploy	Consumers are updated, batch by batch	Both work
4. Validate	Every consumer proves it uses the new value	Both work
5. Retire	The authority stops accepting the old value	Only new works

Step 2 is the whole design. It converts a change that must be atomic into two changes that need not be, and the window between step 2 and step 5 is the overlap period.

Not every authority supports dual acceptance, and where it does not the rotation is a different, harder change:

Supported natively: most identity providers, API gateways with multiple active keys, database users where a second user can be created, JWT signing with a key set rather than a single key.
Not supported: a single shared password on a service that holds exactly one. Here the options are a brief coordinated outage, or introducing an indirection — a second credential — before rotating, so that the next rotation has the overlap this one lacked.

The second option is worth taking the first time the problem appears, because rotation is not a one-off.

`no_log` on everything that touches the value

Read-only / Safewhat no_log does, executed— Executed on ansible-core 2.21.3 with -v. The first task is protected; the second, identical but unprotected, prints the value. Both a scalar result and a loop item are covered.

$ ansible-playbook rotation-demo.yml -v

TASK [Task handling the secret, protected] *************************************
ok: [localhost] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}

TASK [Same task, unprotected] **************************************************
ok: [localhost] => {"ansible_facts": {"rotated2": "REPLACE_ME_NEW"}, "changed": false}

TASK [Loop with no_log] ********************************************************
ok: [localhost] => (item=(censored due to no_log)) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result"}
ok: [localhost] => (item=(censored due to no_log)) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result"}

Note that the loop item is censored as well as the result. That matters, because a loop over per-host credentials would otherwise print each key in the task header regardless of what the result contains.

What no_log does not cover

It suppresses task output. It does not make the play secret-safe on its own.

The value is still on the target. no_log is about the controller’s output, not the file you wrote.
A failing task still reveals context. The message is censored, but the failure and its position are not, and diagnosing a rotation with every task censored is genuinely hard — which is the pressure that produces the next item.
Removing it to debug is how the leak happens. A rotation that fails at 02:00 and gets -vvv with no_log commented out prints the new secret into the scrollback, the terminal recording and the ticket. Debug with a deliberately fake value in a test environment instead.
--diff is separate. A template task writing a credential shows the content unless no_log is on that task specifically.
The value is on the controller. Vault-encrypted at rest, decrypted in memory during the run, and in any fact cache that captured it.

The rotation play

Service impact possibledeploy, batched, with validation between batches— serial keeps the exposure bounded. The validation task proves the consumer works with the new value before the next batch starts.

- name: Rotate the API credential on the consumers
hosts: appservers
serial: [1, 10, 50]
max_fail_percentage: 0
vars:
  api_credential: "{{ vault_api_credential_new }}"
tasks:
  - name: Write the new credential
    ansible.builtin.template:
      src: api-credentials.j2
      dest: /etc/app/credentials
      owner: root
      group: app
      mode: '0640'
    no_log: true
    notify: reload app

  - name: Apply the reload before validating
    ansible.builtin.meta: flush_handlers

  - name: The consumer authenticates with the new credential
    ansible.builtin.uri:
      url: "http://127.0.0.1:8080/healthz/upstream"
      status_code: 200
    register: upstream
    retries: 6
    delay: 5
    until: upstream is succeeded

  - name: The consumer reports the new credential fingerprint
    ansible.builtin.assert:
      that:
        - upstream.json.credential_fingerprint == expected_new_fingerprint
      fail_msg: "{{ inventory_hostname }} still presenting the previous credential"
      quiet: true

Three details.

meta: flush_handlers before validation. Handlers run at the end of the play or batch. Without the flush, the validation task tests a process that has not reloaded yet, and it will pass — against the old credential, which still works because of the overlap. The validation would be measuring nothing.

max_fail_percentage: 0 — any failure stops the rotation. A consumer that cannot authenticate with the new value is a reason to stop and understand why, not to continue and produce more of them.

A fingerprint, not the secret. The consumer reports a hash or a key ID, so the validation proves which credential is in use without any task, log or evidence file containing the credential.

Step 5 — retire, and prove it

The step that gets skipped, because after step 4 everything works.

Skipping it leaves the old value valid indefinitely. A rotation that does not retire the old credential has not rotated anything — it has added a second working credential, which is strictly worse than before, since a compromised value is now valid and nobody is watching it.

Read-only / Safeprove the old value stops working— Run from the controller after retirement. A 401 is the pass condition, which is why status_code lists it explicitly.

- name: The retired credential is rejected
ansible.builtin.uri:
  url: "https://api.example.com/v1/whoami"
  headers:
    Authorization: "Bearer {{ vault_api_credential_old }}"
  status_code: [401, 403]
register: retired_check
no_log: true
delegate_to: localhost
run_once: true

- name: Record the retirement as evidence
ansible.builtin.assert:
  that:
    - retired_check.status in [401, 403]
  fail_msg: "The previous credential is still accepted - retirement did not take"
  success_msg: "Previous credential rejected with {{ retired_check.status }}"

The negative test is the only evidence that the rotation achieved its purpose. Everything before it proves the new value works, which was never in doubt.

Why 'the hosts that were unreachable' is the hardest part

Every rotation has a tail, and the tail is where rotations become permanent problems.

A 300-host rotation typically reaches 294. The other six are rebooting, in a failed state, powered off, or behind a network problem. They hold the old credential, and three things are now true at once:

They still work, because of the overlap. So nothing alerts.
They are absent from the recap, so the run looks complete unless somebody compares against the intended host list — the same denominator problem as an aborted window and an incomplete evidence directory.
They break at retirement, which is hours or days later, by which point the rotation is not the first hypothesis anybody reaches for.

The structural fix is to make retirement conditional on a measured number rather than on elapsed time:

- name: Every host in the intended list has the new credential
  ansible.builtin.assert:
    that:
      - rotation_confirmed_hosts | length == intended_hosts | length
    fail_msg: >-
      {{ (intended_hosts | difference(rotation_confirmed_hosts)) | join(', ') }}
      still hold the previous credential

intended_hosts comes from the inventory, not from the play’s recap. rotation_confirmed_hosts comes from the validation step, which each host contributes to only after proving it uses the new value.

That assertion turns the tail from something nobody tracked into a named list of hosts blocking the retirement — which is a work item somebody can pick up, rather than an incident three days later.

The other consequence worth planning for: the overlap must be long enough for the tail. If hosts are routinely unreachable for two days, a four-hour overlap does not cover the estate you actually have. That number is knowable — it is in the reachability history of previous runs.

What happens if the rotation has to be aborted halfway?

The fleet is split: some hosts on the new value, some on the old. Because of the overlap, both work, so there is no urgency — which is the overlap doing exactly its job.

The response is not to reverse. Reversing means pushing the old value back to hosts that already have the new one, which doubles the work and leaves the same split at a different boundary.

Instead:

Stop, and record the split. Which hosts have which value, from the validation output rather than from an assumption about how far the play got.
Confirm the overlap covers the delay. If the rotation resumes next week, the overlap has to still be open next week — and if it cannot be, the retirement date moves rather than the rotation being rushed.
Do not retire. The most damaging thing available is retiring the old value while part of the fleet still presents it. If retirement is automated on a timer, that timer must be cancelled explicitly, and this is the argument for making retirement a separate manual step gated on the count assertion above.
Resume with the same batching, limited to the hosts that still hold the old value.

The failure to avoid: treating a half-rotated fleet as an emergency because it is untidy, and pushing the rest through without the batching and validation that the first half received. The overlap exists precisely so that a half-rotated fleet is not an emergency.

Knowledge check

Why this matters

Secret rotation is one of the few operations where the standard advice and the standard practice are furthest apart. Every security baseline requires periodic rotation. Most estates rotate rarely, and the reason is that the first attempt caused an outage and the second was scheduled indefinitely.

The outage almost always has the same cause: no overlap. Somebody changed the value at the authority and then pushed it out, because that is the obvious order, and it is the order that guarantees a window in which the fleet is wrong.

Once the dual-acceptance window exists, rotation stops being an event and becomes a routine change that can be batched, validated, paused halfway and resumed next week. The property that makes all of that possible is a single one: at every moment, both values work, so no step in the sequence has a deadline measured in seconds.

The cost of building it is real — it may mean introducing a second credential before the first rotation, which feels like a detour. It is the difference between rotating quarterly and rotating once.

The problem is ordering, and it has no atomic solution

no_log on everything that touches the value

The rotation play

Step 5 — retire, and prove it

Knowledge check

`no_log` on everything that touches the value