Runbook: Add a role

A new role is a new thing that will run against production hosts on every converge, forever, without anyone reading it again. The cost of getting it wrong is not the first run - that one has an audience. It is run four hundred, when the role quietly restarts a service every time because a task is not idempotent, and nobody notices because it has always done that.

This runbook is about getting it right before it becomes furniture.

When to use this runbook

• A new service is being brought under configuration management.
• Repeated task blocks in a playbook are being factored into a role.
• A hand-managed configuration file is becoming declared state.
• A third-party role is being replaced with an internal one.

Blast radius

Zero until Step 10. Steps 1 to 7 touch only the repository and a disposable test container. Step 8 is check mode against one host. Step 9 changes one host. Step 10 is where the role reaches a group, and the size of that group is the number you wrote down in the pre-checks.

Step 1: Scaffold, then delete what you do not need

cd /srv/automation/repo/roles
ansible-galaxy init nginx_frontend
find nginx_frontend -type f | sort

The scaffold creates every standard directory. Delete the ones this role does not use. An empty vars/, files/ and tests/ in every role teaches readers that the layout is decoration rather than information, and the one role that genuinely does have a vars/main.yml stops standing out.

Keep: tasks/, defaults/, meta/, plus handlers/, templates/ and files/ if used.

Step 2: Write the tasks

---
- name: Web server package is installed
ansible.builtin.package:
  name: "{{ nginx_frontend_package }}"
  state: present

- name: Site configuration is rendered
ansible.builtin.template:
  src: site.conf.j2
  dest: "/etc/nginx/conf.d/{{ nginx_frontend_site }}.conf"
  owner: root
  group: root
  mode: '0644'
  backup: true
  validate: 'nginx -t -c %s'
notify: Reload nginx

- name: Web server is running and enabled
ansible.builtin.systemd_service:
  name: nginx
  state: started
  enabled: true

Three things in that snippet are the difference between a role and a liability:

• validate: runs the config checker against the rendered file before it replaces the live one. Without it, a template bug ships a broken config and the handler then fails to reload a service that is now unable to start.
• backup: true leaves a timestamped copy on the host. That copy is your per-host rollback material, and it is the only one that exists if the role was applied outside a change window.
• mode: '0644' is quoted. Verified on 2.21.3: 0644 unquoted is parsed as YAML 1.1 octal, giving 420 - which is 0o644, so it happens to be right. But 644 without the leading zero is decimal 644, which is 0o1204: the sticky bit and permissions nobody intended, with the task reporting success. Quoting is the one form that does not depend on remembering which of those you wrote.

Step 3: Interface in defaults, constants in vars

---
# The role's public, overridable interface.
nginx_frontend_package: nginx
nginx_frontend_site: default
nginx_frontend_listen_port: 8080
nginx_frontend_worker_processes: auto

defaults/ sits at the bottom of variable precedence, which is exactly what makes it the interface: anything in group_vars or host_vars overrides it.

vars/main.yml sits near the top and cannot be overridden from inventory. Put a tunable there and an operator has to edit the role to change it, which they will do, on a branch, and then it stops being shared.

Prefix every variable with the role name. Unprefixed port or config_dir collide across roles, silently, and the collision surfaces as a value that is correct in one play and wrong in the next.

Step 4: Refuse bad input at task zero

---
argument_specs:
main:
  short_description: Configure the front-end web tier
  options:
    nginx_frontend_site:
      type: str
      required: true
    nginx_frontend_listen_port:
      type: int
      default: 8080
    nginx_frontend_upstreams:
      type: list
      elements: str
      required: true

Validation from an argument spec is inserted as a task tagged always and runs at the start of the role. A missing required variable becomes a named failure before anything is touched, rather than an undefined variable error three tasks in, after the package has been installed and the service stopped.

That difference matters most in a rollout: a role that fails at task zero leaves the host exactly as it was, which is a clean batch failure you can stop on.

Step 5: Handlers that actually fire

---
- name: Reload nginx
ansible.builtin.systemd_service:
  name: nginx
  state: reloaded

Prefer reloaded to restarted where the service supports it - a reload keeps connections; a restart drops them, and across a group that is an outage rather than a change.

Step 6: Lint and syntax-check

ansible-lint roles/nginx_frontend
ansible-playbook site.yml --syntax-check
ansible-playbook site.yml --list-tasks | grep -i nginx_frontend

--syntax-check parses; it does not evaluate templates or connect anywhere. --list-tasks is the cheap way to confirm the role is actually reachable from the playbook - a role that was added to the repository but never referenced produces no output here, and that is a common way for a “deployed” role to have never run.

Step 7: Molecule, including idempotence

cd roles/nginx_frontend
molecule test

The default molecule test sequence includes lint, converge, idempotence and verify against a disposable instance. The idempotence step is the one that earns its runtime: it converges twice and fails if the second run reports any change.

Step 8: Check mode against one canary

ansible-playbook site.yml --limit web01.example.com \
--tags nginx_frontend --check --diff | tee check-nginx-web01.txt

Read the diff for:

• Files being replaced that were previously hand-edited.
• Values that rendered as empty strings, which means a variable did not resolve.
• A service state change you did not intend.

Check mode also lies in predictable ways here: validate: does not run in check mode because there is no rendered file to validate, and any task conditioned on a command result is evaluated against a result that does not exist. A clean check run is a necessary gate, not a guarantee.

Step 9: Real run on the canary, then hold

ansible-playbook site.yml --limit web01.example.com \
--tags nginx_frontend --diff | tee converge-nginx-web01.txt

Point of no return for this host. The role has now changed it, and removing the role from the playbook will not put it back.

Verify with something that would notice a real failure:

ansible web01.example.com -b -m command -a 'nginx -t' -o
curl -sS -o /dev/null -w '%{http_code}\n' http://192.0.2.11:8080/healthz

# Idempotence on the real host, which the container could not prove
ansible-playbook site.yml --limit web01.example.com --tags nginx_frontend --diff

That last run must report changed=0. If it does not, stop: you have found a non-idempotent task, and rolling it to the group would mean a service reload on every host on every converge from now on.

Hold the canary long enough to see it under real traffic. An hour is usually enough to catch a configuration that is syntactically valid and operationally wrong.

Step 10: Attach to the group and roll in batches

- name: Front-end web tier
hosts: web
become: true
serial:
  - 1
  - "25%"
roles:
  - role: nginx_frontend
    tags: [nginx_frontend]

ansible-playbook site.yml --limit web --tags nginx_frontend --list-hosts
ansible-playbook site.yml --limit web --tags nginx_frontend --diff

serial: [1, "25%"] runs one host, then quarter-sized batches. Verified on 2.21.3: a percentage that does not divide evenly rounds down to a whole host with a minimum of one, so a six-host group under serial: [1, 2, "30%"] produced batches of 1, 2, 1, 1, 1. Read the batch boundaries in the output rather than assuming them.

Run --list-hosts first, every time. It is two seconds and it is the only thing that catches a --limit that resolves wider than you meant.

Rollback

# The template task took a backup; find it and read the timestamps
ansible web01.example.com -b -m find \
-a 'paths=/etc/nginx/conf.d patterns="default.conf.*"' -o

# Restore the chosen backup, validate, reload. BACKUP is the exact path
# the previous command reported - do not guess the suffix format.
BACKUP=/etc/nginx/conf.d/default.conf.REPLACE_ME
ansible web01.example.com -b -m copy \
-a "src=$BACKUP dest=/etc/nginx/conf.d/default.conf remote_src=true"
ansible web01.example.com -b -m command -a 'nginx -t'
ansible web01.example.com -b -m systemd_service -a 'name=nginx state=reloaded'

What the rollback does not undo: packages the role installed, users or groups it created, services it enabled, and directories it made. Reversing those needs an explicit teardown play written for the purpose. If reversibility matters for this role, write that teardown now, while you still remember what the role does, rather than during the incident.

Record which hosts were converged before the rollback. That set is not recoverable from the reverted playbook, and the next person to read the repository will reasonably assume the role never ran.

Common patterns

Escalation

Escalate when:

• The role would take over a file that a person currently maintains by hand. That is an ownership decision.
• Another role already manages the same file or service. Two roles with different opinions produce a host that changes on every converge and drift that “fixes itself”.
• The role needs privileges the automation account does not have.
• The canary run changed something check mode did not predict. That gap is the finding; do not roll past it because the service came back up.