Lab: Break the template and watch validate refuse

Objective

By the end of this lab you will have rendered a deliberately invalid nginx configuration through a template task and proved, by checksum, that the file on the managed node did not change and the running service was not touched. You will also have run the same broken render without validate: and seen the difference — which is a service that fails to reload and a host that still serves the old config until the next restart, when it stops serving anything.

Architecture

One managed node running nginx from the distribution package, and a role that owns one vhost file.

controller ──ssh──▶ node1  (Debian 12 / Ubuntu 24.04)
                     ├── /etc/nginx/nginx.conf          (package-owned)
                     ├── /etc/nginx/sites-available/lab.conf   <- managed
                     └── /etc/nginx/sites-enabled/lab.conf     -> symlink

Requirements

A controller with ansible-core 2.21.x.
One managed node with systemd as PID 1 and nginx installed from the distribution package. A container is acceptable if you stop at the validator and make no assertion about service state; the second half of the lab — reload versus restart behaviour with a bad config — requires a real service manager. Declare B-nested.
SSH key access and become.
Port 80 free on the node. The lab binds a vhost on 127.0.0.1:8080 rather than :80 so it does not collide with anything already served.
No out-of-band access requirement: this lab does not reconfigure SSH, the firewall or any network interface. It can stop nginx, which Cleanup restores.

Scenario

A colleague deployed a vhost change on Friday. The template rendered a variable that was undefined in production, producing a server_name ; line. nginx -t would have rejected it. The play did not run nginx -t; it wrote the file and notified a reload.

The reload failed, so nginx carried on with its previously-loaded configuration and kept serving correctly. Nobody noticed. Nine days later the node rebooted, nginx failed to start, and the site went down for reasons that appeared entirely unrelated to a change made a week and a half earlier.

Tasks

Task 1: Capture the starting state

WORKDIR="$HOME/ansible-validate-lab"
mkdir -p "$WORKDIR"/{templates,files}
cd "$WORKDIR"

inventory.yml:

web:
  hosts:
    node1:
      ansible_host: 192.0.2.11
  vars:
    ansible_user: operator

# capture.yml
- name: Capture the pre-lab nginx state
  hosts: web
  become: true
  gather_facts: false
  tasks:
    - name: Record which sites are enabled
      ansible.builtin.command: ls -la /etc/nginx/sites-enabled/
      register: sites
      changed_when: false

    - name: Record the service state
      ansible.builtin.command: systemctl is-active nginx
      register: active
      changed_when: false
      failed_when: false

    - name: Confirm the current configuration is valid before we start
      ansible.builtin.command: nginx -t
      register: pretest
      changed_when: false
      failed_when: false

    - name: Save the capture to the controller
      ansible.builtin.copy:
        content: |
          host: {{ inventory_hostname }}
          nginx active: {{ active.stdout }}
          nginx -t rc: {{ pretest.rc }}
          sites-enabled:
          {{ sites.stdout }}
        dest: "{{ playbook_dir }}/pre-lab-nginx.txt"
        mode: '0644'
      delegate_to: localhost
      become: false

Read-only / Safecontroller— Reads service state and configuration validity. Changes nothing.

$ ansible-playbook -i inventory.yml capture.yml

If nginx -t rc is not 0 in the capture, stop. The node’s configuration is already broken and nothing this lab shows you will be interpretable.

Task 2: Write the vhost template

templates/lab-vhost.conf.j2:

# Managed by Ansible. Local edits are overwritten.
# Source: {{ template_path | default('templates/lab-vhost.conf.j2') }}
# Rendered: for host {{ inventory_hostname }}

server {
    listen 127.0.0.1:{{ vhost_port }};
    server_name {{ vhost_server_name }};

    access_log /var/log/nginx/{{ vhost_name }}.access.log;
    error_log  /var/log/nginx/{{ vhost_name }}.error.log;

    root /var/www/{{ vhost_name }};
    index index.html;

    location / {
        try_files $uri $uri/ =404;
    }
}

Note the $uri in the try_files line. Jinja leaves $ alone — it is not a Jinja sigil — so nginx variables pass through untouched. This is one of the few places where nginx and Jinja syntax do not collide.

Task 3: The play, with validation

# deploy.yml
- name: Deploy the lab vhost
  hosts: web
  become: true
  gather_facts: false
  vars:
    vhost_name: lab
    vhost_port: 8080
    vhost_server_name: lab.example.com

  handlers:
    - name: Reload nginx
      ansible.builtin.systemd_service:
        name: nginx
        state: reloaded

  tasks:
    - name: Ensure the document root exists
      ansible.builtin.file:
        path: "/var/www/{{ vhost_name }}"
        state: directory
        mode: '0755'

    - name: Write a placeholder index
      ansible.builtin.copy:
        content: "lab vhost\n"
        dest: "/var/www/{{ vhost_name }}/index.html"
        mode: '0644'

    - name: Render the vhost, validating before activation
      ansible.builtin.template:
        src: lab-vhost.conf.j2
        dest: "/etc/nginx/sites-available/{{ vhost_name }}.conf"
        owner: root
        group: root
        mode: '0644'
        backup: true
        validate: 'nginx -t -c %s'
      notify: Reload nginx

    - name: Enable the vhost
      ansible.builtin.file:
        src: "/etc/nginx/sites-available/{{ vhost_name }}.conf"
        dest: "/etc/nginx/sites-enabled/{{ vhost_name }}.conf"
        state: link
      notify: Reload nginx

`nginx -t -c %s` on a vhost fragment will fail — and that is the first lesson

Run the play. The validation fails, with something like:

nginx: [emerg] "server" directive is not allowed here in /tmp/.../source:6

nginx -t -c FILE treats FILE as a whole nginx configuration, and a server { } block is only legal inside http { }. Your vhost fragment is not a valid standalone config, so the validator correctly rejects it.

This is the single most common mistake with validate: on nginx, and the usual response is to delete the validate: line — which removes the only control that would have caught the real problem.

The fix is to validate the whole configuration with the candidate file in place, which requires a two-step deploy: write to a staging path, validate the full tree, then move it into position.

    - name: Render the vhost to a staging path
      ansible.builtin.template:
        src: lab-vhost.conf.j2
        dest: "/etc/nginx/sites-available/{{ vhost_name }}.conf.candidate"
        owner: root
        group: root
        mode: '0644'
      register: candidate

    - name: Validate the whole configuration with the candidate linked in
      ansible.builtin.shell: |
        set -euo pipefail
        ln -sfn "/etc/nginx/sites-available/{{ vhost_name }}.conf.candidate" \
                "/etc/nginx/sites-enabled/{{ vhost_name }}.conf.candidate"
        trap 'rm -f "/etc/nginx/sites-enabled/{{ vhost_name }}.conf.candidate"' EXIT
        nginx -t
      args:
        executable: /bin/bash
      register: validation
      changed_when: false
      when: candidate.changed

    - name: Promote the candidate only once the whole config validated
      ansible.builtin.copy:
        src: "/etc/nginx/sites-available/{{ vhost_name }}.conf.candidate"
        dest: "/etc/nginx/sites-available/{{ vhost_name }}.conf"
        remote_src: true
        owner: root
        group: root
        mode: '0644'
        backup: true
      when: candidate.changed and validation.rc == 0
      notify: Reload nginx

Which validators can run against a temporary file

validate: '<command> %s' runs the command against a temporary copy, so it only works for tools that accept an arbitrary path and validate it in isolation. Record this table in your notes:

Service	Validator	Works on a temporary path?
sudoers	`visudo -cf %s`	yes — `/etc/sudoers` is self-contained
sshd	`sshd -t -f %s`	yes
nginx (whole config)	`nginx -t -c %s`	yes
nginx (a vhost fragment)	`nginx -t -c %s`	no — needs the enclosing `http {}`
Apache vhost fragment	`apachectl -t -f %s`	no, same reason
chrony	`chronyd -Q -f %s`	yes

The rule: if the file is a complete configuration in its own right, validate: works directly. If it is a fragment included by something else, you need the staged approach above.

visudo -cf %s is the canonical example of validate: and the reason it exists — a broken /etc/sudoers locks every operator out of become on that host, and the tempfile check prevents it absolutely.

Task 4: Break the template deliberately and prove nothing moved

Take a checksum of the live file first. This is the evidence:

ansible -i inventory.yml web -b -m stat \
  -a 'path=/etc/nginx/sites-available/lab.conf checksum_algorithm=sha256' \
  | grep checksum

Record it. Now break the render by leaving vhost_server_name undefined:

ansible-playbook -i inventory.yml deploy.yml \
  --extra-vars '{"vhost_server_name": ""}'

With an empty server name the template produces server_name ;, which nginx rejects:

Configuration changecontroller— The validation refuses. The live file is untouched.

$ ansible-playbook -i inventory.yml deploy.yml --extra-vars '{"vhost_server_name": ""}'

TASK [Validate the whole configuration with the candidate linked in] ***********
fatal: [node1]: FAILED! => {"changed": false, "cmd": "...", "rc": 1,
"stderr": "nginx: [emerg] invalid number of arguments in \"server_name\" directive in /etc/nginx/sites-enabled/lab.conf.candidate:6\nnginx: configuration file /etc/nginx/nginx.conf test failed"}

PLAY RECAP *********************************************************************
node1                      : ok=4    changed=1    unreachable=0    failed=1

Illustrative output

Now prove the claim rather than believing the recap:

# Same checksum as before?
ansible -i inventory.yml web -b -m stat \
  -a 'path=/etc/nginx/sites-available/lab.conf checksum_algorithm=sha256' \
  | grep checksum

# Is the configuration still valid?
ansible -i inventory.yml web -b -m command -a 'nginx -t' 

# Is the service still serving?
ansible -i inventory.yml web -b -m uri \
  -a 'url=http://127.0.0.1:8080/ return_content=yes'

The checksum is unchanged, nginx -t returns 0, and the vhost still answers. The only trace of the failed run is a .candidate file, which the trap in the validation step already unlinked from sites-enabled.

Task 5: Run the same break without validation, and compare

Comment out the validation and promote steps and go back to writing directly to the live path with the original template task, minus validate:.

ansible-playbook -i inventory.yml deploy-unvalidated.yml \
  --extra-vars '{"vhost_server_name": ""}'

Observe the three-part failure:

# 1. The file on disk is now invalid
ansible -i inventory.yml web -b -m command -a 'nginx -t'

# 2. The reload handler failed - but nginx is still running
ansible -i inventory.yml web -b -m command -a 'systemctl is-active nginx'

# 3. And still serving, from the configuration it loaded before
ansible -i inventory.yml web -b -m uri -a 'url=http://127.0.0.1:8080/'

Service impact possiblecontroller— The state the scenario describes: broken on disk, healthy in memory

$ ansible -i inventory.yml web -b -m command -a 'nginx -t'

node1 | FAILED | rc=1 >>
nginx: [emerg] invalid number of arguments in "server_name" directive in /etc/nginx/sites-enabled/lab.conf:6
nginx: configuration file /etc/nginx/nginx.conf test failed

Illustrative output

That is the nine-day gap in the scenario. Every health check passes. systemctl is-active says active. The site responds. And a reboot takes it down.

What could go wrong?

What happens if you restart instead of reload, with a bad config on disk?

The service stops and does not come back. reload asks the running master process to re-read configuration; if the new config is invalid the master refuses and carries on with the old one, so the failure is survivable and invisible. restart kills the master first. There is no old configuration left in memory to fall back to, so the start fails and the port goes dark.

Reload is therefore the safer verb and the one that hides the problem. That combination is exactly why validate: matters more than the choice of verb.

What happens if the validator itself is missing?

validate: 'nginx -t -c %s' on a node without nginx installed fails the task with a command-not-found error. That is the correct behaviour — a gate that cannot run should not be silently skipped — but it means adding validate: to a role that also installs the package requires the install task to come first, in the same play.

What happens if %s is left out of the validate string?

Ansible fails the task with an explicit error rather than running the command against the live file. The %s is how the temporary path is passed; without it there is nothing to validate.

Task 6: Restore and re-prove

ansible-playbook -i inventory.yml deploy.yml
ansible -i inventory.yml web -b -m command -a 'nginx -t'
ansible -i inventory.yml web -b -m systemd_service -a 'name=nginx state=restarted'
ansible -i inventory.yml web -b -m uri -a 'url=http://127.0.0.1:8080/'

The restart is the real test. A node whose config is valid restarts cleanly; the one from Task 5 would not have.

Validation

pre-lab-nginx.txt records nginx -t rc: 0 before the lab began.
With the staged-validation play and a valid render, the vhost is written, enabled and reloaded, and http://127.0.0.1:8080/ returns the placeholder content.
With vhost_server_name empty, the play fails at the validation step, and the sha256 of /etc/nginx/sites-available/lab.conf is identical to the value recorded before the failed run.
After the failed run, nginx -t still returns 0 and the vhost still answers.
With validation removed, the same break leaves nginx -t returning 1 while systemctl is-active nginx still reports active.
After Task 6, nginx -t returns 0 and nginx survives an explicit state: restarted.
Your notes table lists at least four validators and identifies which two cannot run against a temporary path.

Expected Outcome

ansible-validate-lab/
├── deploy.yml
├── deploy-unvalidated.yml
├── inventory.yml
├── pre-lab-nginx.txt
├── templates/lab-vhost.conf.j2
└── validators.md

On the node: a valid nginx configuration, a lab vhost serving on 127.0.0.1:8080, and nginx surviving a full restart. You have a checksum-backed demonstration that a rejected render changed nothing, and you have seen the alternative.

Troubleshooting

nginx: [emerg] "server" directive is not allowed here. Expected on a fragment; this is Task 3’s lesson. Use the staged approach.

The validation step reports changed on every run. It is a shell task that reads. Add changed_when: false — it is one of the honest uses, because nginx -t cannot modify anything.

ln -sfn leaves a stale candidate symlink after a failure. The trap ... EXIT should remove it. If you see one, the shell exited in a way that skipped the trap — check that executable: /bin/bash is set, since sh on Debian is dash and handles set -o pipefail differently.

uri reports Connection refused. nginx is not listening on 8080. Either the vhost is not enabled (ls /etc/nginx/sites-enabled/), or the reload never happened because nothing reported changed.

backup: true created a file you did not expect. That is the point — it writes lab.conf.<timestamp>~ beside the destination before overwriting. Cleanup removes them; in production they are how you get the previous version back without a git checkout.

The play fails with Destination directory /etc/nginx/sites-available does not exist. nginx was installed from a source that does not use the Debian sites-available layout. Adjust the paths to /etc/nginx/conf.d/lab.conf, and note that conf.d files are included by the packaged nginx.conf on RHEL-family systems the same way.

Cleanup

This lab wrote configuration into /etc/nginx, created a document root, and at one point left an invalid configuration on disk. All of it must come back, and nginx must be left both valid and running.

# cleanup.yml
- name: Remove the lab vhost and restore nginx
  hosts: web
  become: true
  gather_facts: false
  vars:
    vhost_name: lab
  tasks:
    - name: Remove the enabled symlink
      ansible.builtin.file:
        path: "/etc/nginx/sites-enabled/{{ vhost_name }}.conf"
        state: absent

    - name: Remove any leftover candidate symlink
      ansible.builtin.file:
        path: "/etc/nginx/sites-enabled/{{ vhost_name }}.conf.candidate"
        state: absent

    - name: Remove the vhost file, the candidate and every backup
      ansible.builtin.shell: |
        set -euo pipefail
        rm -f /etc/nginx/sites-available/{{ vhost_name }}.conf
        rm -f /etc/nginx/sites-available/{{ vhost_name }}.conf.candidate
        rm -f /etc/nginx/sites-available/{{ vhost_name }}.conf.*~
      args:
        executable: /bin/bash
      changed_when: true

    - name: Remove the document root the lab created
      ansible.builtin.file:
        path: "/var/www/{{ vhost_name }}"
        state: absent

    - name: Remove the log files the lab created
      ansible.builtin.file:
        path: "{{ item }}"
        state: absent
      loop:
        - "/var/log/nginx/{{ vhost_name }}.access.log"
        - "/var/log/nginx/{{ vhost_name }}.error.log"

    - name: Validate the restored configuration BEFORE restarting
      ansible.builtin.command: nginx -t
      register: post
      changed_when: false

    - name: Restart nginx only once the configuration validated
      ansible.builtin.systemd_service:
        name: nginx
        state: restarted
      when: post.rc == 0

Verify against the capture:

cd "$HOME/ansible-validate-lab"
cat pre-lab-nginx.txt

ansible -i inventory.yml web -b -m command -a 'ls -la /etc/nginx/sites-enabled/'
ansible -i inventory.yml web -b -m command -a 'systemctl is-active nginx'

The sites-enabled listing must match the pre-lab capture, and nginx must report active. Only then:

rm -rf deletes the whole lab directory, including anything you added

The next block removes ~/ansible-validate-lab and everything under it, recursively and without confirmation. If you saved notes, an inventory you want to keep, or a playbook you extended, copy it out first — the step above does that for the lab’s own deliverables and knows nothing about yours.

The path is written in full and anchored to $HOME deliberately. A relative rm -rf run from the wrong directory is how people delete work they meant to keep, and the shell gives no second chance.

Confirm what you are about to delete before you delete it:

ls -la "$HOME/ansible-validate-lab"

mkdir -p "$HOME/ansible-lab-deliverables"
cp -a validators.md templates/lab-vhost.conf.j2 \
      "$HOME/ansible-lab-deliverables/"

rm -rf "$HOME/ansible-validate-lab"

Under the hood

template and copy do not write to the destination path. They render or stage the content into a temporary file inside the remote temp directory, run validate: against that path if one is given, and then move the temporary file into place with an atomic rename.

Two properties follow, and both matter operationally.

The validation runs before anything reaches the destination, so a failed validation cannot leave a partial or invalid file — there is nothing to clean up because nothing was written. This is why the checksum in Task 4 is unchanged rather than merely “restored”.

And the final step is rename(2) within the same filesystem, which is atomic. A process reading the destination sees either the whole old file or the whole new one, never a truncated mixture. That is why a service reading its configuration during a deploy cannot observe a half-written file — and also why dest: on a different filesystem from the remote temp directory degrades to a copy, losing the atomicity. remote_tmp in ansible.cfg controls where that temporary lives.

%s is substituted with the temporary path, not the destination. A validator that resolves relative includes against its own directory — which is exactly what nginx does with include sites-enabled/* — will therefore resolve them against the temp directory, which is the mechanism behind the fragment problem in Task 3.

What You Learned

validate: on a config fragment fails for a real reason. A vhost is not a standalone nginx configuration, and nginx -t -c %s correctly rejects it. The answer is to stage and validate the whole tree, not to delete the gate.
A rejected render leaves the destination byte-identical. You proved it by sha256, not by reading the recap.
Without validation, the failure is invisible. Broken config on disk, a failed reload, systemctl is-active reporting active, and the site still answering — until a reboot.
reload survives a bad config and restart does not. The safer verb is also the one that hides the damage, which is why the gate matters more than the verb.
validate: works on complete configurations and not on fragments. You have a table of four validators and know which two need staging.
Cleanup validates before restarting, for the same reason deployment does: deleting from a live configuration directory is a change like any other.

Objectives

Prerequisites