Schedules, workflows and unattended runs · Ansible

Everything in this part so far has kept a human in the loop. Somebody decided to run something and watched the output.

Schedules remove the human. That is the entire value and the entire risk, and the two are not separable.

The easy case: scheduled jobs that only look

Start with the class of scheduled automation that is straightforwardly a good idea, because it establishes the pattern before the arguments start.

Scheduled job	Why it wants a schedule	What it changes
Compliance validation	The answer is only useful if it is current	Nothing
Certificate expiry check	Expiry is a date, and dates arrive on their own	Nothing
Patch-level reporting	“How far behind are we” is a weekly question	Nothing
Drift detection with `--check`	Divergence accumulates silently	Nothing
Fact collection into a cache	Inventory questions want fresh facts	Nothing on the fleet

Every row changes nothing. That is what makes them easy: the worst case for a read-only scheduled job is a wrong answer or no answer, and both are recoverable on Monday.

Read-only scheduled automation is the highest-value, lowest-risk thing in this part

If you take one operational recommendation from this lesson, take this one: schedule the checking, not the fixing.

A nightly --check --diff run against production, with the diff kept, is one of the most valuable artefacts an estate can have. It tells you every morning what has drifted, what a convergence run would do, and — the part people miss — whether your automation still works, before the day you need it in an emergency.

It cannot break production, because it changes nothing. Note the qualifier from Part XXV that still applies: a clean --check is not proof that a real run is safe, and modules without check-mode support report nothing meaningful. As a drift signal it is excellent; as a safety proof it remains what Part XXV said it was.

The hard case: scheduled change

Now the version that gets argued about. Nightly convergence — the schedule that applies the desired state and corrects anything that has drifted.

The case for it is genuinely strong. Drift is corrected before it compounds. The gap between the repository and reality never grows large enough to be frightening. And every run exercises the automation, so it does not rot.

The case against is one sentence: production is being modified while nobody is watching.

Concretely, what “nobody is watching” costs:

Failures are discovered late. A convergence run that starts failing at 03:00 on Sunday is discovered on Monday morning, if somebody reads the right dashboard. Between those two points, drift accumulates unchecked and the estate believes it is converged.

Failures are discovered as a batch. By the time anyone looks, it has failed nine times, and the ninth failure is the only one anybody investigates. The first failure is the one that would have explained it.

A change lands with no observer. The run corrects the drift, restarts the service, and the restart fails on eleven hosts. Nobody is on the terminal. The alert that fires is a service alert at 03:14 from a monitoring system, and the on-call engineer starts debugging a service outage without knowing that automation touched it four minutes ago.

Ownership evaporates. A run somebody launched belongs to that person for the next hour. A run a schedule launched belongs to whoever notices.

The 03:14 problem

That third one is worth dwelling on, because it is the failure that converts a routine automation defect into a long incident.

The on-call engineer has a service alert and no reason to suspect a change. They begin at the service — logs, resource usage, recent deployments — and the recent-deployments check comes up empty, because the change was a convergence run, not a deployment.

The time between “service alert” and “somebody thinks to check the automation history” is frequently thirty minutes and occasionally hours. It is entirely avoidable, and the fix is not more careful automation:

Every scheduled job that changes anything must emit a start and finish notification into the same channel your service alerts arrive in. Not a dashboard, not an email folder. The channel where an engineer at 03:14 is already looking.

The point is not that anybody reads it at 03:14. The point is that when they scroll up, the answer is four lines above the alert.

Deciding what to schedule

The useful test is not “is this playbook safe”. It is:

What is the worst thing this can do to a host? Reporting, no change, restart a service, reboot, replace data. Each rung up the ladder needs a stronger reason to be unattended.
How many hosts can one firing reach? A schedule with no --limit fires against the whole inventory every time, forever. This is the hosts: line question with the operator removed.
Who finds out, how fast, when it fails? If the answer is “the dashboard, eventually”, it is not ready to be scheduled.
Does it need a change window, and does the schedule know about windows? A cron expression does not know your business.

A staged path to scheduled convergence

The teams that run nightly convergence successfully almost always got there in this order:

Alert-only. Schedule --check --diff. Publish the diff. Do nothing else, for several weeks. This is where you discover that your automation would have changed forty things nobody expected.

Converge one tier, narrowly. Add --limit to the least critical group. Real changes, small blast radius, a notification into the operational channel.

Widen, slowly, with serial. A converging schedule should carry the same batching a human-run rollout carries, and max_fail_percentage so a systemic failure stops rather than proceeding through the fleet at 03:00.

Only then, everything. And keep the --check schedule running alongside, because it is your drift signal and it is independent of whether the convergence job is working.

The step people skip is the first one, and it is the one that finds the surprises while they are still free.

The cron equivalent, and the mistake in it

You do not need a platform to schedule automation. You need cron or a systemd timer, and you need to not make the standard error.

Here is the standard error:

0 3 * * * ansible-playbook -i /srv/ansible/inventory/production /srv/ansible/converge.yml

Two problems, and the second is the one that hurts.

The first is that the output goes to cron’s mail, which on most modern hosts goes nowhere at all.

The second is that the exit code is discarded. Ansible is careful about exit codes and this line throws all of that away:

Read-only / Safewhat a scheduled run reports when the fleet is unreachable— Verified on ansible-core 2.21.3 against hosts in 192.0.2.0/24, which are unreachable by design. The task itself is read-only; the run never reached a host.

$ ansible-playbook -i inventory.ini compliance.yml; echo "exit=$?"

PLAY RECAP *********************************************************************
web01.example.com          : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0
web02.example.com          : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0
web03.example.com          : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0

exit=4

Exit 4 — unreachable. Nothing in the cron line above notices. The job “ran” every night for three weeks and reached nothing, and the only evidence is a file nobody opened.

A defensible wrapper is not much longer:

#!/usr/bin/env bash
# jobs/nightly-compliance.sh - runs from a systemd timer at 03:00.
set -uo pipefail

LOG=/var/log/ansible/compliance-$(date +%Y%m%d-%H%M%S).log
cd /srv/ansible

ansible-playbook -i inventory/production/hosts.ini playbooks/compliance.yml \
  --limit webservers > "$LOG" 2>&1
rc=$?

case "$rc" in
  0) notify "compliance run clean" ;;
  2) notify "compliance FAILED tasks (rc=2), see $LOG" ;;
  4) notify "compliance UNREACHABLE hosts (rc=4), see $LOG" ;;
  *) notify "compliance run exited $rc, see $LOG" ;;
esac
exit "$rc"

Note set -uo pipefail without -e: with -e the script would exit at the ansible-playbook line and never reach the notification, which is the opposite of what a scheduled job needs. Capturing rc immediately after the command is the whole point.

Distinguishing 2 from 4 matters more than it looks. Failed tasks mean your automation or your hosts have a problem. Unreachable means the run never happened — a different incident, with a different first investigation, and the one most likely to be silently long-running.

Workflows

A workflow template chains job templates into a graph, where each node branches on the previous node’s result: on success, on failure, or always.

The shape that justifies the feature:

[ pre-flight checks ]
        |
   on success
        |
[ canary: one host ]
        |
   on success ------------------> on failure
        |                              |
[ rollout: remaining hosts ]     [ collect diagnostics ]
        |                              |
   on failure                     [ notify + stop ]
        |
[ collect diagnostics ]

That is the canary pattern from Part XXX, expressed as objects rather than as a paragraph in a runbook — with the property that the branch is taken by the system rather than by a person deciding whether the canary looked alright.

Approval nodes pause a workflow until a human with permission approves. This is the mechanism that makes an unattended workflow acceptable at a higher risk level: check and canary run unattended, and the fleet rollout waits for a person.

Surveys parameterise a launch — a form whose answers become extra variables. Everything lesson 2 said about constraining the target set applies here with more force, because a survey is the mechanism by which somebody who has never read the playbook supplies input to it.

Why the failure branch is the one that has never run

Every workflow has a branch nobody has tested, and it is always the same one.

The success path runs daily. It is exercised, observed, and fixed when it breaks. The failure path runs when something has already gone wrong — which is rare, and which is precisely the moment nobody wants to discover that the diagnostics job references a credential that was rotated in March.

The mechanism of the rot is worth naming, because it explains why this happens even to careful teams. A failure branch typically:

uses a different credential (a read-only diagnostic account),
targets a different inventory (the hosts that failed, or a bastion),
calls a job template nobody else calls, so no other run keeps it warm,
and produces output nobody reads when it does work.

Four independent reasons for silent decay, all of them invisible in a green dashboard.

Two practices fix it, and both are cheap:

Run the failure branch on purpose. Once a quarter, launch the diagnostics template directly. Not the whole workflow — just the node. If it fails, it was going to fail during the incident.

Make the failure branch dull. Collect logs, notify, stop. A failure branch that attempts remediation is a second untested change executing during a situation you do not yet understand, which is the amplification problem lesson 7 is about, arriving early.

What happens if a scheduled patching job fires during a Sev-1?

It makes the incident worse, and it will happen eventually, because cron does not know about your incident.

The pattern is grimly reliable: an incident starts at 02:40, the patching schedule fires at 03:00, and now the responders are debugging a system that is changing underneath them. Worse, they may not know it is changing, because the schedule is normal and nobody thinks to mention it.

Three mitigations, in ascending order of effort and effectiveness.

A precondition in the play. The first task checks an incident-suppression flag — a file, an API, a variable in a group_vars file — and refuses if it is set. This uses the guardrail machinery from Part XXIV and is the version available to everyone, platform or not. Its weakness is that somebody has to remember to set the flag during an incident, which is when people remember the least.

A maintenance-window condition. Schedule the job to fire only in windows the business has agreed to, and make the play refuse to run outside its window regardless of how it was launched. This narrows the overlap rather than closing it, since incidents happen during change windows too.

Disabling schedules as an incident-response step. In a platform, this is disabling the schedule or the job template. Without one, it is systemctl disable --now ansible-converge.timer or commenting the cron entry. Either way the requirement is the same and it is procedural: the incident checklist has a line for it, near the top, and somebody owns executing it.

The third is the only one that reliably works, because it does not depend on the automation being clever. It depends on the incident procedure knowing that scheduled automation exists — which is why the list of what is scheduled belongs in the incident runbook and not only in the platform.

Knowledge check

Why this matters

Scheduling is where automation stops being a tool you use and becomes a system that runs your estate.

That transition is worth making. It is the difference between drift being corrected and drift being noticed, between a compliance answer that is current and one that is six months old, between automation that is exercised daily and automation that is discovered to be broken during an emergency.

But it changes what you are responsible for. An automation you run is something you watch. An automation that runs itself is something you have to instrument, because the feedback loop that used to be a person reading a terminal now has to be built out of notifications, exit codes and dashboards that somebody actually reads.

The practical division is durable and worth remembering long after the product details fade. Schedule the looking freely — it is nearly all upside. Schedule the changing carefully, in stages, with batching, with notification into the channel where humans already are, and with a documented way to turn it off that appears near the top of your incident checklist.

And when you are unsure which side something falls on, ask the question that has run through this whole part: if this fires at 03:00 on a Sunday and does the wrong thing, how many hosts does it reach before anyone knows?