Skip to main content
RunBook Academy

← All break/fix scenarios in Ansible

intermediatetemplating~30 min

Break/Fix: the payment gateway drops connections at 02:07 every morning and no change is scheduled

Reported symptoms

  • ●In-flight requests to the payment gateway fail at 02:07 every morning, seven days a week
  • ●The failure lasts under two seconds and is invisible on any dashboard with a one-minute resolution
  • ●No deployment, release or configuration change is scheduled at that time
  • ●The change-management calendar is empty for the window
  • ●The nightly Ansible converge runs at 02:00 and reports success every night
  • ●The converge reports `changed=1` every night, on a task nobody has edited in months

Evidence

  • · `journalctl -u payment-gw --since -7d | grep -c Stopped` returns 7
  • · The service restart timestamps line up with the converge to within a few seconds
  • · The run log shows the same template task reporting `changed` on every host on every run
  • · `ansible-playbook ... --check --diff --limit <host>` shows a one-line difference in a comment header
  • · The difference is a timestamp, and it differs from the previous run by exactly the run interval
  • · `grep -n ansible_date_time roles/payment/templates/gateway.conf.j2` finds the generated-on header
  • · Every other line of the rendered file is byte-identical between runs
Diagnosis and resolutionclick to reveal

Root cause

The template renders a comment header containing the time it was generated. Ansible decides whether a templated file changed by comparing the fully rendered content against what is on the target, so a header carrying a timestamp guarantees a difference on every run. The task therefore reports `changed` every night, truthfully, and the `notify` on that task fires the restart handler every night. Nothing about the service configuration has actually changed since the header was added; the only thing that differs is the record of when the file was written. The obvious diagnosis is wrong twice over: the restart looks like a scheduled job, so the investigation starts at cron and systemd timers, and once Ansible is identified the task looks like a legitimate change, because `changed` is an accurate report of what the task did. The defect is that `changed` no longer means anything useful - a file whose content includes the current time can never be idempotent, so the signal that is supposed to distinguish "the configuration moved" from "the configuration is stable" has been permanently pinned to the first value.

Remediation

Remove the timestamp from the rendered content. If a provenance header is wanted - and it is a reasonable thing to want - it must carry only values that change when the configuration changes: the role name, the repository, the commit. Anything derived from the current time, the run identifier, or a random value belongs outside the file. Then decouple the restart from the template so a future accident is less expensive: the handler should reload rather than restart if the service supports it, and the play should drain the host from the load balancer before either. Deploy the corrected template to one canary host, confirm two consecutive converges report no change, and only then roll out.

Verification

Converge twice with no change in between and require `changed=0` on the second run for the template task specifically, not merely for the play. Confirm no handler section appears in the second run, and confirm from `journalctl -u payment-gw` that no restart occurred. Prove the check can fail by editing a real configuration value on a test host and confirming the next converge reports `changed=1`, notifies, and restarts - a template that never reports changed is a different and worse fault. Finally, watch the 02:00 window for three consecutive nights and require zero restarts, because a fault that fires once a day needs more than one night to be shown absent.

Prevention

Never render a value into a managed file that changes independently of the configuration. Timestamps, run identifiers, random values, and any fact that is re-measured every run all break change detection permanently. Make the two-converge check a CI gate so a template that cannot settle fails before it is deployed. Treat a service restart as an event that must have a cause: if a restart happens on a schedule rather than in response to a change, the change detection feeding it is broken and every other handler in that role is suspect too. Prefer reload to restart where the service supports it, and put drain and health-check steps around anything that interrupts connections. Alert on unexplained service restarts as their own signal, since a restart with no change behind it is either this fault or a crash loop, and both are worth knowing about.

Reported symptoms

The payments team has a recurring complaint that has been open for five weeks. Once a night, at almost exactly 02:07, a handful of in-flight requests to the payment gateway fail with connection resets. Volume at that hour is low, the blip lasts under two seconds, and the error budget absorbs it - which is why it took five weeks to become somebody’s problem.

What the investigation found first:

  • No cron job runs at that time on those hosts.
  • No systemd timer fires in that window.
  • The change-management calendar is empty.
  • No deployment pipeline is scheduled overnight.
  • The service is not crashing: journalctl shows a clean stop and a clean start, not a fault.

The nightly Ansible converge runs at 02:00 and is green every night. Nobody looked at it for a fortnight, on the grounds that it had not changed and it was not deploying anything.

Evidence provided

Read-only / Safea clean restart, once a night, near the converge window
$ ansible pay01 -i inventory -b -m ansible.builtin.command -a 'journalctl -u payment-gw --since -7d -o short-iso | grep -E "Stopped|Started" | tail -6'
2026-08-09T02:07:11+0000 systemd[1]: Stopped payment-gw.service
2026-08-09T02:07:12+0000 systemd[1]: Started payment-gw.service
2026-08-10T02:07:04+0000 systemd[1]: Stopped payment-gw.service
2026-08-10T02:07:05+0000 systemd[1]: Started payment-gw.service
2026-08-11T02:07:19+0000 systemd[1]: Stopped payment-gw.service
2026-08-11T02:07:19+0000 systemd[1]: Started payment-gw.service
Read-only / Safethe converge is the caller
$ grep -E 'TASK|RUNNING HANDLER|changed:' logs/converge-2026-08-11.log | head -8
TASK [payment : Render the gateway configuration] ******************************
changed: [pay01]

RUNNING HANDLER [payment : restart payment-gw] *********************************
changed: [pay01]
Read-only / Safeone line differs, and it is not configuration
$ ansible-playbook -i inventory site.yml --limit pay01 --check --diff --tags payment
TASK [payment : Render the gateway configuration] ******************************
--- before: /etc/payment-gw/gateway.conf
+++ after: /etc/payment-gw/gateway.conf
@@ -1,4 +1,4 @@
# Managed by Ansible - do not edit by hand
-# Generated 2026-08-11T02:07:03Z
+# Generated 2026-08-11T14:22:41Z
# Role: payment

changed: [pay01]
Read-only / Safethe header
$ head -4 roles/payment/templates/gateway.conf.j2
# Managed by Ansible - do not edit by hand
# Generated {{ ansible_date_time.iso8601 }}
# Role: payment
Read-only / Safethe commit, five weeks ago
$ git log --oneline -1 -- roles/payment/templates/gateway.conf.j2
8b1c04e docs: add a provenance header to generated config files

Work the evidence before reading on

Everything in the chain is behaving exactly as designed. The task reported changed because the file changed. The handler ran because it was notified. The service restarted because that is what the handler does.

  1. Read the --check --diff output and count the lines that differ. Then ask which of them is configuration.
  2. ansible_date_time is a fact. When is it measured, and how often does its value repeat?
  3. The service restarts at 02:07 but the converge starts at 02:00. What accounts for the seven minutes, and does it tell you anything about which task is responsible?

Before continuing: if you rendered this template twice in a row with no other change, would the two outputs be identical? What does that answer imply about every run, forever?

Root cause

1. The rendered content includes the current time

ansible.builtin.template decides whether it has anything to do by rendering the template with the current variables and comparing the result against the file on the target. If they match, it reports ok. If they differ, it writes the new content and reports changed.

The provenance header renders ansible_date_time.iso8601, a fact gathered at the start of each run. Its value is different on every run by construction. The rendered content is therefore different on every run by construction, and the comparison can never come out equal.

The task is not wrong about what it did. It genuinely wrote a different file. It is the file that is wrong, in the specific sense that it contains a value which is not part of the configuration.

2. notify inherits the defect

notify fires when a task reports changed. Once changed is pinned to true, the handler is pinned to running.

The restart is not a bug in the handler, in systemd, or in the service. It is the accurate downstream consequence of a change signal that has stopped carrying information. Every handler in this role has the same problem for the same reason, and so does every drift report, every compliance snapshot, and every dashboard that counts changed hosts.

3. The header was a good idea implemented in the wrong place

The commit that caused this is entirely sympathetic. Knowing when and by what a file was generated is genuinely useful when you are staring at an unfamiliar host at three in the morning.

The mistake is the specific value. Provenance that identifies what generated the file - the role, the repository, the commit - is stable between runs and changes exactly when the configuration changes. Provenance that records when it was generated changes every time and takes change detection down with it.

Resolution

  1. Suppress the nightly restart tonight while you work, either by pausing the converge or by temporarily removing the notify. Five weeks of a nightly outage does not need a sixth.
  2. Remove the timestamp from the template. Keep the provenance if it is wanted, but express it with values that change only when the configuration changes - the role, the repository, the deploying commit.
  3. Render on a canary host and compare against the existing file to confirm the only difference is the header you just corrected. There should be no surprises hiding underneath five weeks of noise.
  4. Converge the canary twice. The second run must report changed=0 for the template task and must produce no handler section.
  5. Change the handler from restart to reload if the service supports reloading. A restart drops connections; a reload usually does not, and this handler will still fire on genuine changes.
  6. Add drain and health-check steps around whatever the handler does, and batch the play so the tier does not lose all its capacity at the same instant.
  7. Roll out, then confirm across three consecutive nights that no restarts occur. A once-a-night fault needs more than one night of silence to be considered gone.
  8. Audit the rest of the repository for the same header. It was added by a commit that touched several roles, and every one of them has been reporting changed ever since.

Verification

  1. The second converge reports no change for that task specifically. Read the task line in the output, not just the play recap - another task in the same play may legitimately be changing something and would mask this.
  2. No handler runs on the second converge. The output contains no RUNNING HANDLER section, and journalctl -u payment-gw records no stop or start at that timestamp.
  3. The check can fail. Edit a genuine configuration value on a test host, converge, and confirm the task reports changed, notifies, and reloads. A template that never reports changed is the opposite fault and is worse.
  4. Three consecutive nights are clean. journalctl -u payment-gw --since -3d | grep -c Stopped returns 0 on a sample of hosts.
  5. Client-visible errors have stopped. Check the payments error rate in the 02:00 to 02:15 window across those three nights - the service being quiet and the client being happy are different claims.
  6. The rendered file is stable across hosts and runs. Render on two hosts and diff; anything that differs and is not host-specific configuration is a latent instance of the same fault.
  7. The audit is complete. grep -rn ansible_date_time roles/*/templates/ returns nothing, and any role that used the same header has been fixed and two-converge tested.

Prevention

  • Nothing measured per run belongs in managed content. If the value changes when the configuration has not, it is not configuration.
  • Gate the two-converge check in CI. Render, converge, converge again, and fail the build on any changed in the second pass. This catches the entire family before deployment and costs one extra run.
  • Treat an unexplained restart as a broken change signal until proven otherwise. A service that restarts on a schedule rather than in response to a change is telling you that something upstream of the handler cannot settle.
  • Prefer reload to restart wherever the service supports it, and put drain and health-check steps around anything that interrupts connections. The handler will fire correctly one day, and that is the day it matters that it fires safely.
  • Remember that handlers flush together, so a whole batch restarts at once. Batching the play is what turns a correct handler into a safe one.
  • Put provenance in the file if you want it, but make it identify the producer, not the moment. A commit hash is more useful at three in the morning than a timestamp anyway, because it tells you what to go and read.