Where validate is not enough · Ansible

validate: is the right default and it has three limits. Knowing them is what separates using it from trusting it.

Limit one: the validator needs the whole tree

nginx -t -c %s parses the temporary file as a complete configuration. That works when the file you are templating is the whole configuration. It does not work when you are templating a fragment that is included by a larger file.

A drop-in under /etc/nginx/conf.d/ contains a bare server { ... } block. Handed to nginx -t -c on its own, that is not a valid nginx configuration at all — a server block must live inside http. The validator rejects a file that is perfectly correct in its real position.

The mirror image is worse: a fragment can be individually valid and break the assembled configuration. Two drop-ins that each define server_name www.example.com are both fine alone and produce a conflict warning together. validate sees one file and cannot know about the other.

The same shape appears everywhere:

Configuration	The fragment problem
`/etc/nginx/conf.d/*.conf`	`server` blocks outside `http`
`/etc/sudoers.d/*`	`visudo -cf` handles these well — a genuine exception
`/etc/systemd/system/.d/.conf`	Drop-in overrides are meaningless without the base unit
`/etc/haproxy/conf.d/*`	Backends referenced from a frontend in another file

Limit two: valid and wrong

A configuration can parse perfectly and be operationally incorrect. This is the one that produces the confusing outage.

proxy_pass http://192.0.2.99:8080; is impeccable nginx. nginx -t approves it. The reload succeeds. And if nothing is listening on that address, every request returns 502.

The validator’s contract is parseable, not correct. It checks grammar. It does not check that the backend exists, that the certificate matches the hostname, that the port is the one the application listens on, or that the upstream group is not empty because a hostvars lookup came back empty under --limit.

Limit three: there is no offline checker

Plenty of services have none. A validator does not exist for most application configuration files, many agent configs, and anything whose “check” requires connecting to something.

Some services offer a partial one. Some offer a check that starts the service, which is not a check. And for a JSON or YAML file, the useful check is often structural rather than semantic:

Configuration changea structural validator where no real one exists— python3 -c parses the staged file and exits non-zero on malformed JSON. It proves the file is well formed, not that the settings are right.

- name: Deploy the application configuration
ansible.builtin.template:
  src: app.json.j2
  dest: /etc/app/app.json
  owner: root
  group: app
  mode: '0640'
  validate: python3 -c "import json,sys; json.load(open(sys.argv[1]))" %s
notify: app config changed

That is worth having. A template that renders a trailing comma into a JSON file is a real and common defect, and this catches it. Be clear with yourself about what it proves.

Building the sequence by hand

When validate: cannot do the job, the sequence it implements can be built explicitly: render to a staging path → check → activate → notify.

Service impact possiblerender, check, activate, notify— The template writes to a staging path. The validator runs against the full assembled tree. Only then is the file moved into the live directory.

- name: Deploy a validated nginx drop-in
hosts: webservers
become: true
tasks:
  - name: Render the drop-in to a staging path
    ansible.builtin.template:
      src: site.conf.j2
      dest: /etc/nginx/staging/site.conf
      owner: root
      group: root
      mode: '0644'
    register: staged

  - name: Validate the whole configuration with the drop-in included
    ansible.builtin.command:
      argv:
        - /usr/sbin/nginx
        - -t
        - -c
        - /etc/nginx/nginx.staging.conf
    changed_when: false
    when: staged.changed

  - name: Activate the validated drop-in
    ansible.builtin.copy:
      src: /etc/nginx/staging/site.conf
      dest: /etc/nginx/conf.d/site.conf
      remote_src: true
      owner: root
      group: root
      mode: '0644'
    when: staged.changed
    notify: Reload nginx

handlers:
  - name: Reload nginx
    ansible.builtin.systemd_service:
      name: nginx
      state: reloaded

Four design decisions in that play, each of which is the difference between it working and it being a liability.

register: staged and when: staged.changed. Without these, the validator runs on every host on every run and the activation task runs on every run. Guarding both on the staging render’s own changed result means a converged host does nothing at all — which is the same conditional discipline handlers use, applied one level down.

changed_when: false on the validator. It is a read-only probe. Let it report changed and it contaminates the recap and, if you ever add a notify nearby, the handler chain.

remote_src: true on the activation. The staged file already exists on the managed node; without this, copy would look for the source on the controller.

notify on the activation task, not the render. The render happens before the check. Notifying there would queue a reload for a configuration that has not been validated yet — and if a later task fails, force_handlers would then apply it. The notification belongs on the task that puts the file in the live path.

A staging file is a new failure mode

The hand-built sequence introduces something validate: does not have: a second copy of the configuration, in a directory, on every host, forever.

Three things to get right:

Keep the staging directory outside the include glob. A staging path under /etc/nginx/conf.d/staging/ will be picked up by include conf.d/*.conf on some layouts, at which point you have activated the unvalidated file by leaving it there. Use a path the service does not read.
Permissions apply to the staging copy too. A 0640 secret staged at 0644 is a 0644 secret, in a file nobody is looking at.
The staging file persists. After a failed validation it sits there holding the broken content, and the next run’s staged.changed will be false because the file already matches — so the activation is skipped and the failure is not repeated. That is a silent never-activated state, and it is the reason the next section exists.

Keeping the sequence honest

The trap above is worth stating on its own, because it is the failure this pattern actually produces in the field.

Run 1: the template renders new content to staging, reports changed, validation fails, the task errors, the play stops for that host. Run 2: the template renders the same content to staging, which already matches, so it reports ok. staged.changed is false. The validator is skipped. The activation is skipped. The play is green and the configuration was never activated.

That is the never-fired-handler failure from Part XVI, reconstructed from the same conditional habits that were supposed to prevent it.

Two ways to close it, and the second is better:

Service impact possibleguard on the destination, not the staging file— Compare the staged content against the live file rather than against the previous staged file. A never-activated change stays visible on every subsequent run.

- name: Render the drop-in to a staging path
ansible.builtin.template:
  src: site.conf.j2
  dest: /etc/nginx/staging/site.conf
  mode: '0644'

- name: Read the checksum of the staged and live files
ansible.builtin.stat:
  path: '{{ item }}'
register: files
loop:
  - /etc/nginx/staging/site.conf
  - /etc/nginx/conf.d/site.conf

- name: Validate and activate when the live file differs from the staged one
when: files.results[0].stat.checksum != files.results[1].stat.checksum | default('')
block:
  - name: Validate the assembled configuration
    ansible.builtin.command:
      argv: [/usr/sbin/nginx, -t, -c, /etc/nginx/nginx.staging.conf]
    changed_when: false

  - name: Activate the validated drop-in
    ansible.builtin.copy:
      src: /etc/nginx/staging/site.conf
      dest: /etc/nginx/conf.d/site.conf
      remote_src: true
      mode: '0644'
    notify: Reload nginx

The condition now asks the question that actually matters — does the live file differ from what we intend? — rather than did we write a new staging file this run? A change that failed validation on run 1 is still pending on run 2, gets validated again, and fails again. Loudly, every run, until somebody fixes it.

Under the hood

The manual sequence is reconstructing what the module does internally, and comparing the two is instructive about what you lose.

The module stages inside the destination directory and finishes with atomic_move, a rename within the same filesystem, so the replacement is instantaneous from a reader’s point of view. The manual sequence uses a second copy task, which is also atomic for the same reason — but only if the staging path and the destination are on the same filesystem. Stage under /tmp on a host where /tmp is a separate mount and the copy becomes a read-and-write rather than a rename, reopening the partial-write window the module closed.

The module also holds the staged file for the duration of one task. The manual sequence holds it forever, which is the difference between a temporary file and a piece of state you now own.

And the module’s guard — “only do any of this if the checksums differ” — is computed against the destination, not against a previous staging file. That is exactly the correction the second example above applies, and it is not a coincidence: guarding on the live file rather than on intermediate state is the general principle. Intermediate state can agree with your intent while the thing you actually care about does not.

What happens if the assembled configuration is valid but a different drop-in is broken?

Your validation fails, on a change that has nothing to do with you.

That is the honest cost of validating the whole tree: the check is no longer scoped to the file you are managing. A colleague’s broken drop-in, hand-edited on the host during an incident and never cleaned up, will fail your role on that host and block a change that was fine.

It is still the right trade. A validator that only checks your fragment would have approved a configuration the service will reject on reload, which converts a failed Ansible task into a failed service start.

What it argues for is the drop-in discipline in the next lesson, and for noticing the shape of the failure: a validation error naming a file your role does not manage is a signal that the host has been edited by hand, which is a drift problem rather than a template problem.

Knowledge check

Why this matters

The previous lesson gave you a line of YAML that converts the most likely template defect into a failed task. This one is the honest footnote: it catches unparseable, it does not catch wrong, and it cannot see a fragment in context.

Both halves matter. A team that does not know about validate: ships broken configuration to production. A team that believes validate: means the change is safe ships a syntactically flawless configuration pointing at a backend that does not exist, then spends the outage looking at nginx rather than at the variable that produced the address.

And when you do have to build the sequence yourself, the thing that will bite you is not the validation. It is the conditional: a guard on intermediate state that quietly stops re-checking a change that never got activated. Guard on the live file. It is the only thing whose state you actually care about.