Break/Fix: every health check passed, every task succeeded, and the checkout has been broken for six hours

Reported symptoms

The release went out at 02:00. At 08:15 the support queue makes it obvious that nobody has been able to complete a purchase since 02:06.

Every system involved says everything is fine:

• The Ansible run is green. Twenty-two tasks, zero failures, including a task named “Health check after deploy”.
• The rehearsal with --check the previous afternoon was green too.
• The load balancer reports all six backends healthy and in service.
• The uptime dashboard is at 100 percent.
• The application processes are running and are not restarting.

Six hours of failed checkouts, and the first place anybody looked was the payment provider, because nothing in the deployment chain gave any reason to suspect the deployment.

Evidence provided

$ grep -B1 -A2 'Health check' logs/rehearsal-check.log

TASK [Health check after deploy] ***********************************************
skipping: [node01]
skipping: [node02]
skipping: [node03]

$ ansible-doc ansible.builtin.uri | grep -A3 'check_mode:'

        check_mode:
description: Can run in check_mode and return changed status prediction without modifying
target, if not supported the action will be skipped.
support: none

$ grep -n -A8 'Health check after deploy' deploy.yml

44:- name: Health check after deploy
45:  ansible.builtin.uri:
46:    url: "http://{{ inventory_hostname }}/"
47:    status_code: 200
48:  retries: 6
49:  delay: 5
50:  register: health
51:  until: health.status == 200

$ curl -s -o /dev/null -w '%{http_code} %{time_total}s\n' http://node01/

200 0.004s

$ curl -s -o /dev/null -w '%{http_code}\n' -H 'Cache-Control: no-cache' http://node01/checkout/health

500

$ ansible node01 -i inventory -b -m ansible.builtin.command -a 'tail -2 /var/log/app/error.log'

2026-08-11T02:06:11Z ERROR checkout: configuration key payment.gateway_url is not set
2026-08-11T02:06:11Z ERROR checkout: returning 500

Work the evidence before reading on

Three separate checks passed and one of them was named as the health gate. Work out, for each, what question it was able to answer.

1. In the rehearsal the health-check task says skipping. What does check-mode support of none mean for a task under --check?
2. The root returns 200 in four milliseconds. What component answered, and did the application take part?
3. The load balancer says every backend is healthy. What does its probe request?

Before continuing: is there any state of the application in which this health check could have returned something other than 200? If not, what was it measuring?

Root cause

1. The rehearsal skipped the gate entirely

--check asks each module to predict what it would do. Modules that cannot meaningfully predict - because they act on the outside world rather than on the target’s state - declare no check-mode support, and Ansible skips them.

uri, wait_for and wait_for_connection are all in that category. Under --check they do not run, and the output says skipping.

In a run of twenty-two tasks with several legitimate skips from conditionals, one more skipping line reads as nothing at all. The rehearsal was interpreted as “the deployment is safe, including the health gate”, and what it actually demonstrated was that every other task would behave.

The tasks most worth rehearsing before a release are the gates, and those are precisely the tasks check mode cannot rehearse.

2. The gate asserted something that is always true

The live run did execute the gate, and the gate passed, because it asked a question with only one possible answer.

The request goes to the site root. The root is a static page. The reverse proxy on each node serves it from cache in four milliseconds without consulting the application at all. It would return 200 with the application stopped, with the database unreachable, and with every configuration key in the file deleted.

The status code was checked and the body was not, so even a cached error page with a 200 status would have satisfied it.

3. The load balancer has the same defect, which is why nothing alerted

The load balancer probe also requests the root. So does the uptime monitor.

Three independent-looking checks agreed, and their agreement was convincing precisely because they were not independent - all three asked the same weak question of the same cache. A single strong check would have been worth more than the three of them combined.

Resolution

1. Restore the checkout path first. Whether that is a roll back or a configuration fix rolled forward depends on how the key was lost, but customers come before diagnosis.
2. Confirm from outside the network that checkout works, with a cache-bypass header, before declaring the incident over. Every check that said things were fine was made from inside.
3. Rewrite the gate to exercise the checkout path, bypass the cache, and assert on the response body as well as the status code.
4. Prove the new gate fails. Point it at a deliberately broken backend and require the play to fail. Until you have watched it reject something, you have replaced one unverified claim with another.
5. Fix the load balancer probe in the same change. It has the same defect, and it is the reason six hours passed with no alert.
6. Add a standalone gate-rehearsal play that runs the health check against the current production version without deploying anything. This is the rehearsal --check cannot give you.
7. Audit the rest of the playbook for tasks that are skipped under --check, and note in the runbook which parts of the play a rehearsal does and does not cover.
8. Work out why the configuration key was missing in the first place. The gate failing to catch it is the incident; the key going missing is a second defect and it has its own cause.

Verification

1. The gate rejects a broken backend. Break one deliberately on a staging node and require the play to fail. This is the only check in this list that could have prevented the incident, and it had never been run.
2. The checkout endpoint responds correctly from outside, with a cache-bypass header, and the response body contains the expected content. Status codes alone are what got everybody here.
3. The load balancer removes a broken backend. Break one on staging and confirm the probe marks it down within its configured interval.
4. The rehearsal covers what you think it covers. Run the deployment with --check and list every task reported as skipping; each one is a part of the play the rehearsal did not exercise, and that list belongs in the runbook.
5. The standalone gate rehearsal passes against production as it stands. If it does not, the gate is wrong before any release is involved.
6. The three checks now disagree usefully. With one backend deliberately broken, the deployment gate, the load balancer probe and the monitoring check should all report a problem; if any of them still reports health, it is asking the weak question.
7. The application error rate for the checkout endpoint is zero, measured at the application rather than at the proxy.

Prevention

• Every gate must be able to fail, and must have been observed failing. Break something on purpose on a schedule, not once at implementation time.
• Health checks exercise the code path the change touched. A generic liveness probe answers a generic question, and releases are specific.
• Bypass caches and proxies deliberately. Anything between the check and the application can answer on its behalf, and a cache is very good at making a dead system look healthy.
• Assert on content, not only on status. A cached error page, a maintenance page and a working application can all return 200.
• Know which modules have no check-mode support, and read skipping in a rehearsal as information. The tasks check mode cannot run are usually the ones the release depends on.
• Do not treat a --check run as a rehearsal of the gates. Rehearse gates separately, against the current version, as their own play.
• Make the deployment gate, the load balancer probe and the monitoring check ask different questions. Three checks that share a weakness give three times the confidence and none of the coverage.