Break/Fix: the new TLS configuration is on disk on every host and no host is serving it · Break/Fix · Ansible

Reported symptoms

A compliance scan flags the entire web tier for offering a cipher the organisation retired three days ago. The change to retire it was deployed three days ago.

The deployment history is short and, on the face of it, reassuring:

Tuesday 14:10 - the play ran, wrote the new TLS configuration, and then failed on a later task that manages a logrotate fragment. 200 hosts failed.
Tuesday 14:40 - the logrotate task was fixed and the play was run again. Completely green. failed=0. The TLS task reported ok.
Wednesday and Thursday - the nightly converge ran green both nights.

The file on every host contains the new cipher policy. The team read ok on the TLS task as “already correct, nothing to do”, which is what it means.

Restarting nginx by hand on one host fixes that host instantly, which made the first hypothesis a caching problem at the load balancer.

Evidence provided

Read-only / Safea cipher the new policy forbids, still negotiated

$ openssl s_client -connect web01.example.com:443 -tls1_2 -cipher 'ECDHE-RSA-AES128-SHA' </dev/null 2>/dev/null | grep -m1 Cipher

    Cipher    : ECDHE-RSA-AES128-SHA

Read-only / Safethe file was written on Tuesday

$ ansible web01 -i inventory -b -m ansible.builtin.command -a 'stat -c %y /etc/nginx/conf.d/tls.conf'

2026-08-08 14:11:52.318 +0000

Read-only / Safethe process has been running since July

$ ansible web01 -i inventory -b -m ansible.builtin.command -a 'systemctl show nginx -p ActiveEnterTimestamp'

ActiveEnterTimestamp=Mon 2026-07-20 03:14:07 UTC

Read-only / Safethe first run - notice what is absent

$ grep -E 'TASK|RUNNING HANDLER|changed:|ok:|fatal:' logs/tuesday-1410.log | head -12

TASK [nginx : Deploy the TLS cipher policy] ************************************
changed: [web01]
TASK [logging : Install the logrotate fragment] ********************************
fatal: [web01]: FAILED! => {"msg": "Destination directory /etc/logrotate.d/vendor does not exist"}

PLAY RECAP *********************************************************************
web01                      : ok=14   changed=1    unreachable=0    failed=1

Read-only / Safethe second run - green, and the reload is still absent

$ grep -E 'TASK|RUNNING HANDLER|changed:|ok:' logs/tuesday-1440.log | head -12

TASK [nginx : Deploy the TLS cipher policy] ************************************
ok: [web01]
TASK [logging : Install the logrotate fragment] ********************************
changed: [web01]

PLAY RECAP *********************************************************************
web01                      : ok=16   changed=1    unreachable=0    failed=0

Work the evidence before reading on

The handler is fine. It is the same handler that reloads nginx successfully in a dozen other plays.

Search both run logs for the string RUNNING HANDLER. How many times does it appear across the two runs combined?
In the first run the TLS task reported changed and the play then failed. What happens to a notification that has been queued but not yet flushed?
In the second run the TLS task reported ok. What does a task that reports ok notify?

Before continuing: is there any future run of this playbook, with no further edits, that would reload nginx? Justify the answer from how notify is triggered.

Root cause

1. Handlers are queued, and the queue does not survive a failure

notify does not run anything. It marks a handler as pending for that host. Pending handlers run at the flush point, which by default is the end of the play.

When a host fails a task, Ansible stops processing that host and its pending handler queue is discarded. Nothing is logged about the discarded handlers, and nothing in the recap distinguishes a host that failed with an empty queue from a host that failed with a reload waiting in it.

The first run wrote the file, queued the reload, and then failed on the logrotate task. The reload was dropped on all 200 hosts.

2. The second run made it permanent

This is the part that turns a recoverable situation into an invisible one.

By 14:40 the file on disk was already correct - the first run wrote it before failing. So the TLS task compared the intended content against the file, found them identical, and reported ok.

A task that reports ok does not notify. There was no notification to queue, no handler to flush, and a perfectly green run.

Every run since has done the same thing, and every run in the future will, because the file will keep matching. The system has settled into a stable state that no rerun can leave:

	Intended	Actual
File on disk	new policy	new policy
Running process	new policy	old policy
Ansible report	`ok`	`ok`

Ansible is not wrong. It manages files, and the file is correct. It has no visibility into what the running process loaded, and never claimed any.

3. `changed=0` was read as a success signal

The team’s reading - “the TLS task says ok, so the change is applied” - is the most natural reading available and it is the wrong one. changed=0 means the described state matches the observed state for everything Ansible describes. Whether a daemon has re-read a file it opened weeks ago is not something Ansible describes.

Under the hood

The behaviour is straightforward to reproduce, and worth reproducing once so the shape is familiar.

A play with a task that reports changed and notifies a handler, followed by a task that fails, produces this recap and no handler section at all:

TASK [Deploy config] ***********************************************************
changed: [localhost]

TASK [Unrelated later task fails] **********************************************
fatal: [localhost]: FAILED! => {"msg": "logrotate fragment missing"}

PLAY RECAP *********************************************************************
localhost   : ok=1  changed=1  unreachable=0  failed=1

The same play run with --force-handlers runs the handler after the failure:

fatal: [localhost]: FAILED! => {"msg": "logrotate fragment missing"}

RUNNING HANDLER [reload nginx] *************************************************
ok: [localhost]

PLAY RECAP *********************************************************************
localhost   : ok=2  changed=1  unreachable=0  failed=1

Note what force_handlers does and does not do. It runs handlers that were already notified on a host that later failed. It does not run handlers on a host that never notified them, and it does not help at all with the second run, where the notification never existed.

That is the important asymmetry: force_handlers protects you at the moment of failure. Once a subsequent run has reported ok, no handler-related setting can recover the situation, because there is nothing to recover.

Resolution

Confirm the fault on more than one host before acting. Compare the file mtime against the service start timestamp on a sample; where the file is newer, the process has not read it.
Reload the service on the affected hosts as a deliberate, rate-limited operation. The configuration is already in place, so this is a reload rather than a deployment - but 200 simultaneous reloads of a TLS-terminating tier is an outage of its own, so batch it.
Verify from a client between batches. Negotiate against a repaired host and confirm the retired cipher is refused; the file has been correct for three days and proves nothing.
Add force_handlers: true to the play, so a later failure can no longer discard a pending reload.
Add an explicit meta: flush_handlers immediately after the configuration block, so the reload happens before any task that could fail rather than at the end of the play.
Add a verification task that asserts the service is serving the intended policy. This is the task that would have failed the green run on Tuesday, and its absence is the reason nobody noticed.
Write down, in the runbook for this play, what to check after a run that failed partway: which tasks reported changed, which of those carry a notify, and therefore which hosts have unflushed handlers.

Verification

A client cannot negotiate the retired cipher. openssl s_client against a repaired host fails for the forbidden cipher and succeeds for a permitted one. This is the only check that is independent of both the file and Ansible.
The running configuration matches the file. nginx -T on the host shows the new policy; nginx -T renders what the process loaded, unlike reading the file.
The service is younger than the file. systemctl show nginx -p ActiveEnterTimestamp is more recent than the file mtime on every repaired host. This comparison is cheap, scriptable across the fleet, and would have caught the fault on Tuesday evening.
The guard can fail. On a test host, write a modified policy and suppress the reload, then run the play: the verification task must fail. A verification that has only ever passed is untested.
A clean rerun passes both conditions together. The play reports changed=0 and the verification task passes in the same run - the broken state satisfies the first condition alone, which is exactly why one is not enough.
Handlers survive a failure now. Reintroduce a deliberately failing task after the TLS block on a test host and confirm the reload still runs, thanks to force_handlers.
The whole fleet is covered. The file-newer-than-service comparison returns nothing across the estate, not just across the hosts you repaired first.

Prevention

Set force_handlers: true on any play whose handlers restart or reload something that matters. The default behaviour is a reasonable safety choice for handlers that should not run on a half-configured host, and a poor one for a reload that was already earned.
Flush explicitly. meta: flush_handlers after the configuration block puts the reload where you can reason about it, instead of at the end of a play that may not get there.
End every service-affecting play with an assertion about the service. A negotiated cipher, a version endpoint, a header, a systemctl show timestamp - anything that can only be true if the daemon re-read the file.
Never treat changed=0 as proof of correctness. It says the files match. It says nothing about what any running process has loaded.
After any run that fails partway, enumerate the hosts with notified but unflushed handlers before rerunning. That information exists only in the failed run log and the rerun erases it.
Compare service start time against configuration file mtime as a fleet-wide drift check. It is one command per host and it finds this entire class of fault, including the cases where a human edited a file and forgot to reload.

What if the first run had failed before the configuration task instead of after it?

Then nothing would have been deployed, the failure would have been obvious, and the fix would have been the ordinary one: repair the failing task and rerun.

The damage in this incident comes entirely from the ordering. The configuration task succeeded and the play died afterwards, leaving the host in a state that is neither the old configuration nor the new one - new on disk, old in memory - and, crucially, a state that a rerun reports as correct.

That ordering hazard generalises well beyond handlers. Any play that performs a change and then verifies or activates it in a later step has a window in which a failure leaves the change half-applied, and any such play needs to answer the question “what does a rerun do from here”. The good answers are: the rerun completes the work, or the rerun refuses. The bad answer, and the one here, is that the rerun reports success.

The related case worth planning for is a host that goes unreachable between the change and the flush. force_handlers cannot help - there is no connection to run the handler over - so the host is left in the same split state with no log entry describing it. That is one of the reasons the mtime-versus-service-start comparison earns its place as a standing check rather than as an incident-time trick.

Reported symptoms

Evidence

Root cause

Remediation

Verification

Prevention

Reported symptoms

Evidence provided

Work the evidence before reading on

Root cause

1. Handlers are queued, and the queue does not survive a failure

2. The second run made it permanent

3. changed=0 was read as a success signal

Resolution

Verification

Prevention

3. `changed=0` was read as a success signal