Break/Fix: the config passed validation, the reload succeeded, and the site returns 502 · Break/Fix · Ansible

Reported symptoms

At 11:04 a routine change adds one backend server to an upstream pool. At 11:06 the site returns 502 for every request.

The deploy looked textbook:

TASK [nginx : Render the upstream configuration] *******************************
changed: [web01]

TASK [nginx : Validate and reload] *********************************************
ok: [web01]

RUNNING HANDLER [nginx : reload nginx] *****************************************
changed: [web01]

PLAY RECAP *********************************************************************
web01                      : ok=22   changed=2    unreachable=0    failed=0

The service is running. It did not crash. It reloaded cleanly. And a task named “Validate and reload” reported success.

The change was reverted at 11:11 and the site recovered, which everybody read as proof that the change was at fault. Three engineers then spent an hour looking at a two-line diff that adds one entry to a YAML list, and could not find anything wrong with it.

They were right. There is nothing wrong with it.

Evidence provided

Read-only / Safethe rendered file - note the missing port numbers

$ ansible web01 -i inventory -b -m ansible.builtin.command -a 'nginx -T' | grep -A5 'upstream app'

upstream app_backend {
server 192.0.2.31:;
server 192.0.2.32:;
}

Read-only / Saferun by hand, it fails immediately

$ ansible web01 -i inventory -b -m ansible.builtin.command -a 'nginx -t'

nginx: [emerg] invalid port in upstream "192.0.2.31:" in /etc/nginx/conf.d/upstream.conf:2
nginx: configuration file /etc/nginx/nginx.conf test failed
non-zero return code

Read-only / Safethe validator - compare this with the module documentation

$ grep -n -A7 'Validate and reload' roles/nginx/tasks/main.yml

31:- name: Validate and reload
32:  ansible.builtin.template:
33:    src: upstream.conf.j2
34:    dest: /etc/nginx/conf.d/upstream.conf
35:    validate: /usr/sbin/nginx -t
36:  notify: reload nginx

Read-only / Safethe template

$ cat roles/nginx/templates/upstream.conf.j2

upstream app_backend {
{% for u in app_upstreams %}
  server {{ u.host }}:{{ u.pot | default('') }};
{% endfor %}
}

Read-only / Safethe change everyone blamed, and the change that actually did it, eleven months apart

$ git log --oneline -3 -- roles/nginx/templates/upstream.conf.j2 group_vars/web.yml

a91f2b7 web: add app-02 to the upstream pool
6d0c115 nginx: make the upstream port optional for the health checker
2f88e41 nginx: initial upstream template

Work the evidence before reading on

The rendered file is wrong in a way that nginx -t catches instantly when a human runs it. A task whose entire purpose is to run nginx -t reported success.

Read the template’s port expression character by character. Compare the key name against the key name in group_vars.
Read the validate string, then read what the module documentation says about how the candidate file is passed to the validation command.
The upstream pool had one entry until yesterday and the site was fine. If the port has been rendering as empty for eleven months, why did the site work?

Before continuing: which file did nginx -t actually examine when Ansible ran it?

Root cause

1. `default` turned a typo into an empty string

The template renders {{ u.pot | default('') }}. The key in group_vars is port. There is no pot.

Without the filter, Jinja would raise an undefined error and the task would fail:

fatal: [web01]: FAILED! => {"msg": "Task failed: ... 'dict object' has no attribute 'pot'"}

That failure is exactly what you want. It is loud, it names the expression, and nothing is written.

With | default(''), the missing key renders as nothing. The line becomes server 192.0.2.31:; - syntactically wrong, semantically meaningless, and produced by a task that reports changed and moves on.

The filter was added eleven months ago in a commit whose message says “make the upstream port optional for the health checker”. That is a reasonable requirement. The implementation applied a default to a field that is not optional at all, and did so on the same line as a typo that the default then concealed.

2. `validate` without `%s` checked the wrong file

This is the defect that made the first one survivable for eleven months and then fatal.

validate on template and copy runs a command against a temporary copy of the newly rendered content, before the file is moved into place. The temporary path is substituted wherever %s appears in the command string. The module documentation states that %s must be present.

The task says:

validate: /usr/sbin/nginx -t

No %s. So nginx -t ran with no file argument, which makes it test the configuration currently installed on the host - the previous, working file. It passed. It would have passed no matter what the template rendered, because the candidate was never shown to it.

For eleven months this task reported ok while checking a file it was not deploying.

3. Why the site worked until yesterday

With a single upstream entry, nginx tolerated the malformed line well enough for the pool to function in this configuration - the fault was latent, not absent. Adding the second entry crossed the threshold where the block could no longer be parsed at all, and the reload brought up a configuration that could not route.

That is why the change looked guilty. It was the trigger, not the cause, and reverting it restored service without touching either defect. The next converge would have redeployed the same broken template.

Under the hood

Three checks were available in this pipeline and none of them could have caught this. It is worth knowing exactly what each one covers, because the team believed they were covered.

--syntax-check parses the playbook YAML and the task structure. It does not render templates, does not resolve variables, and does not contact a host. It passes on this playbook and always would.

--check runs the modules in prediction mode. template supports check mode fully and would have reported that it was going to change the file. It predicts the change accurately; it does not judge whether the change is correct.

--diff is the one that shows the actual defect, and it did, in a run that reported changed alongside dozens of other lines:

+    server 192.0.2.31:;

A malformed port is visible in that line if you read it. Nobody reads diff output carefully on a green run, which is precisely why the validate option exists - it is the mechanism that reads the diff for you and refuses.

Everything hinges on validate being wired correctly, and there is no check anywhere in Ansible that a validate command is well formed.

A validation step that has never rejected anything is not a validation step

The task ran nginx -t on every deploy for eleven months and passed every time. That record was read as evidence that the configuration was always fine. It was evidence that the check never had teeth.

Any guard - a validator, an assertion, a health check, a linter in CI - has to be shown to fail before it can be trusted to pass. The test is cheap: break something on purpose, once, and watch the guard reject it.

For validate specifically, the test is to render a deliberately invalid file on a scratch host and confirm two things: the task fails, and the live file is unchanged. The second half is the whole point of the option. Validation happens against a temporary copy so that a rejected configuration never reaches the running service.

Resolution

Restore service, but understand what the rollback did. Reverting the upstream addition removed the trigger and left both defects in place; the template on disk in the repository still renders a broken file.
Fix the validator first. Add the %s placeholder so the command runs against the rendered candidate: validate: /usr/sbin/nginx -t -c %s or the form appropriate to how your configuration is included. Until this works, no other fix can be verified.
Confirm the fixed validator rejects the current template. Run the play against one scratch host with the uncorrected template and require the task to fail. A validator that now catches the known-bad case is a validator.
Correct the template. Use the right key name, and remove the default filter from the port entirely, since a required field should fail loudly when it is missing.
Audit the rest of the repository for the same two patterns: validate without %s, and default applied to a field that is not optional. Both were introduced by ordinary, well-intentioned commits and both are likely to appear more than once.
Roll forward on a canary host with the corrected template, confirm nginx -T shows complete upstream entries, and fetch a real endpoint from a client before proceeding.
Add an assertion to the play that requests a real endpoint and requires a successful status code, so a future broken configuration fails the run rather than the customer.
Re-apply the original upstream change once the canary is proven, since that change was always correct and is still wanted.

Verification

The validator can reject. Render a deliberately invalid configuration on a scratch host and confirm the task fails. This is the single most important check here and it had never been performed.
A rejected configuration does not reach the host. In that same test, confirm the live file is byte-identical to what it was before, and that the service was not reloaded.
The rendered configuration is complete. nginx -T on a repaired host shows every upstream entry with both an address and a port; grep for a colon followed by a semicolon and require no matches.
The service parses its own configuration. nginx -t run by hand on a repaired host succeeds.
A client gets a real response. Fetch an endpoint and require a successful status code from outside the host. The service running and the service working are different claims, and this incident is the difference between them.
The end-of-play assertion can fail. Point it at a deliberately wrong port on a test host and confirm the run fails.
The repository is clean of both patterns. A grep for validate: lines lacking %s returns nothing, and every remaining default filter has been reviewed against whether the field is genuinely optional.

Prevention

Use default only where the field is genuinely optional, and say so in a comment. On a required field it converts a loud, precise, harmless error into a silently malformed file.
Consider failing fast on undefined variables in templates for configuration you own, so a typo is an error rather than an empty string. The strictness is the feature.
Always include %s in validate, and read the module documentation when you write one rather than copying a command line that works in a shell. The two contexts pass files differently.
Test every guard by breaking something. A validator, assertion or health check that has never rejected anything is an unverified claim.
Assert the outcome, not the step. Green tasks, a successful reload and a running process are all compatible with a service that returns 502 to every request.
Render templates in CI against representative variable sets and check the structure of the output, so a malformed render is caught before a host sees it.
When a small change appears to cause a large failure, check whether it is the trigger rather than the cause. Reverting a trigger restores service and leaves the cause in place for the next converge.

What if the template had raised an undefined-variable error instead of rendering empty?

That is the good outcome, and it is worth seeing what it looks like so the instinct to suppress it can be resisted.

Without the default filter, the task fails on the first host with a message that names the expression and the missing attribute, and nothing is written to any host. The play stops, the site keeps serving, and the fix is a one-character correction to a key name found in under a minute.

The instinct when that error appears in the middle of a deploy is to add a default and move on, because the error looks like an obstacle rather than a finding. That instinct is how this defect was created eleven months ago.

There is a middle path worth knowing. mandatory makes the requirement explicit and produces a message you choose:

{{ u.port | mandatory }}

And when a field really is optional, the template should say what the absence means rather than emitting an empty string into a structured file:

{% if u.port is defined %}    server {{ u.host }}:{{ u.port }};
{% else %}    server {{ u.host }};
{% endif %}

The distinction is between “this value may be absent” and “the output may be malformed”. Only the first is ever an acceptable design.

Why this matters

Two defects, eleven months apart, neither of which is unusual and both of which passed review. A default filter added to make something optional, and a validation command copied from a shell where it works perfectly well.

The reason the combination is worth studying is that each one disabled the check that would have caught the other. The default suppressed the error the template would have raised; the missing %s suppressed the error the validator would have raised. What remained was a green run deploying an unusable configuration to a production tier, and a change that had nothing wrong with it taking the blame.

The habit that closes this is to distrust guards until they have refused something. Every validator, assertion and health check in a pipeline should have a recorded instance of rejecting a deliberately broken input. Without that, a passing check and an absent check look exactly the same.

Reported symptoms

Evidence

Root cause

Remediation

Verification

Prevention

Reported symptoms

Evidence provided

Work the evidence before reading on

Root cause

1. default turned a typo into an empty string

2. validate without %s checked the wrong file

3. Why the site worked until yesterday

Resolution

Verification

Prevention

1. `default` turned a typo into an empty string

2. `validate` without `%s` checked the wrong file