The complete pipeline · Ansible

Six lessons have introduced gates one at a time. This one puts them in order and makes the two decisions that turn a list of tools into a pipeline: what each stage is for, and what happens when it fails.

The organising rule is the one this part has used throughout. Every stage must be able to name the class of production failure it prevents. A stage with no answer to that is not a safety measure; it is a tax on every change, and it will eventually be removed by someone who is right to remove it.

The pipeline, annotated

Ordered cheapest first, because a stage that takes two seconds should never wait behind one that takes six minutes.

#	Stage	Typical time	What it catches that the previous cannot	Production failure it prevents
1	`yamllint`	< 5s	Formatting the parser accepts but humans misread	Review noise hiding a scope change
2	`ansible-playbook --syntax-check`	< 10s	A playbook that cannot be assembled: bad keyword, missing role, missing import	A run that dies at 02:00 having done half the estate
3	Secret scan	10–30s	A credential in the diff or the history	A disclosed credential and a rotation incident
4	Vault-encryption gate	< 5s	A `vault.yml` that is plaintext, or carries the wrong vault id	Production secrets readable by everyone with dev credentials
5	`ansible-inventory --graph` per environment	< 10s	Broken inventory, unparseable group, wrong vault id	A deploy that fails at variable load against production
6	`ansible-lint --profile production`	30–90s	Semantic defects: non-idempotent commands, `latest` packages, missing FQCN	A task that reports `changed` forever, restarting services nightly
7	Molecule, path-filtered	2–10 min	Run-time behaviour: does the role work from nothing, and twice	A role that appends a line per run, or fails on a fresh host
8	`--check --diff` against staging	1–5 min	What this change would actually do, on real hosts, with this environment’s variables	A change whose effect nobody could see before it landed
9	Human review with that evidence	—	Intent, scope, rollback, timing	Correct automation aimed at the wrong machines

Stage 5 is the one people leave out and it is worth its ten seconds. It is the only static stage that loads and decrypts group_vars, so it is the only one that fails on a wrong vault id — --syntax-check exits 0 in that case, measured on 2.21.3.

Stages 1 to 6 are the ones you can express as a script and run identically on a laptop and in CI, which is what stops a pipeline failure from being a surprise:

Read-only / Safescripts/static-gates.sh — every stage that touches no host— Nothing here connects to a managed node. Anything after stage 6 does, and belongs in a job with credentials.

#!/usr/bin/env bash
set -euo pipefail

yamllint -s -f parsable .

for env in development staging production; do
ansible-playbook -i "inventories/$env" playbooks/site.yml --syntax-check
done

scripts/scan-secrets.sh
scripts/check-vault-encrypted.sh

for env in development staging production; do
ansible-inventory -i "inventories/$env" --graph > /dev/null
done

ansible-lint --profile production

The vault ids for stages 2 and 5 come from the runner’s environment; note that the loop deliberately covers every environment, because an inventory that only breaks for production is the one that breaks at the worst moment.

Blocking or reporting

Not every gate should stop a merge, and getting this wrong in either direction is expensive.

Block on: syntax check, secret scan, vault-encryption gate, inventory load, lint at the chosen profile, and Molecule for the roles the change touches. These are all deterministic — they fail because the change is wrong, not because the world moved.

Report on: the check-mode diff. Its job is to produce evidence, and it depends on a staging environment that can be down for reasons unrelated to the change. Blocking on it means a staging outage blocks every merge in the repository, which teaches everyone to use the override. Attach the output, mark the stage failed if it errored, and let the human decide whether they have enough to review.

Report on: anything flaky, until it is not. A gate that fails spuriously once a week trains people to re-run rather than read, and a re-run habit applies equally to the real failures.

The two ways this pipeline was green while production broke

Both causes of this part’s break/fix are visible in the table above, and both are configuration rather than code.

The lint profile excluded the rule that would have caught it. Stage 6 ran, passed, and reported nothing — because the rule was in skip_list, added eighteen months earlier to unblock an urgent merge. A gate is only as strong as its rule set, and the rule set does not appear in the pass line. Lesson 3 covers the warn_list versus skip_list distinction and the expiry-date discipline.

The check gate ran against an inventory that no longer resembled production. Stage 8 ran against a staging inventory that had lost a group months earlier, so the play matched nothing, produced an empty diff, and exited 0. Lesson 5 covers the host-count assertion and the structural-equivalence check that catch it.

Neither failure is detectable from the pipeline’s own output. Both are detectable by periodically asking, of each stage, what did this actually examine — which is a review of the pipeline configuration, on a schedule, by a person. Put it in the same quarterly slot as the exclusion-list expiry review.

Making it fast enough not to be bypassed

The pipeline competes with an alternative that is always available: merging without it. Every minute it costs makes that alternative more attractive, and the moment it wins is an incident.

Four things buy the most time back:

Path filtering. A change under roles/webserver/ runs the webserver Molecule scenarios and nothing else. This is the single biggest win in the whole pipeline and it is a few lines of configuration.

Parallel stages. Stages 1 through 6 are independent of one another. Run them concurrently and the static portion costs as long as its slowest member rather than their sum.

Cached dependencies. ansible-galaxy install -r requirements.yml against a network on every job is both slow and a source of failures unrelated to the change. Cache it keyed on the file’s hash — which works correctly precisely because the versions are pinned.

Scheduled full runs. The complete platform matrix, every role, every scenario, nightly against the default branch. It catches the cross-role and upstream-image regressions that path filtering misses, without putting hours into the merge path.

Under the hood

The degradation path is the part almost nobody designs, and it is what decides whether the pipeline survives its first bad week.

There will be a morning when a stage is broken for reasons that have nothing to do with any change — the container registry is rate-limiting, the staging environment is down, an upstream image changed its base. On that morning somebody also needs to ship an urgent security fix.

Design the answer in advance, because the answer improvised at 09:00 is “disable the stage”, and disabled stages stay disabled.

Tier the gates explicitly. A small set that must pass for anything to merge — syntax, secret scan, vault gate — and which are fast and have no external dependencies. Everything else is a second tier that can be declared degraded.

Make degradation a state, not an action. A flag that marks a stage degraded, with an owner and an expiry, and which is visible on every pull request merged while it is set. That is very different from someone editing the pipeline definition, because it is reversible, attributable and self-expiring.

Compensate rather than skip. A merge that goes through with Molecule degraded should carry more human attention, not the same amount: a canary host, a smaller batch, a named person watching. The pipeline is one input to the safety of a change and it is not the only one available.

Alert on the degradation itself. A gate that has been degraded for three weeks is a gate that has been deleted, and it should page somebody long before that. The expiry is what makes this automatic rather than dependent on someone remembering.

What happens if you are adding this to a repository that has none of it?

Do not build the whole table on day one. A pipeline that turns red for four hundred pre-existing reasons gets switched off in a week, and the switching-off is the durable outcome.

The order that works, roughly one stage a week:

Syntax check, blocking. It passes on almost any repository that works at all, so it goes green immediately and establishes that the pipeline exists and is trusted.
Secret scan, blocking, on the diff only. Scanning the full history of an inherited repository will find things, and you should look — but as a separate remediation project, not as a gate that blocks every merge from day one.
ansible-lint, reporting, at whatever profile it currently passes. Then raise the profile one rung at a time. Part XXV’s adoption lesson covers this in detail; the pipeline-specific part is that it reports rather than blocks until the repository is clean at the target profile, and flips to blocking in the same change that clears the last finding.
The vault and inventory gates. Cheap, fast, and they almost always pass — unless something is genuinely wrong, in which case you wanted to know.
Molecule for one role. Pick the role that changes most often and whose breakage hurts most. One scenario, path-filtered. Extend from there.
The check-mode gate, reporting. It needs a staging environment that resembles production, which may itself be a project. Add it when staging is worth running against.

Each step goes in on its own, green, and stays green. What you are protecting is the pipeline’s credibility: a red build has to mean something, and it only means something if it has never meant “the repository was like this when we found it”.

Knowledge check

Why this matters

A pipeline is a claim about what cannot reach production, and like any claim it is worth exactly as much as the evidence behind it.

The version of this that fails is a long list of stages nobody can justify, running for twenty minutes, bypassed during every incident, with an exclusion list that has been growing quietly since 2024. It is green every day and it prevents almost nothing.

The version that works is shorter than you expect. Every stage names the production failure it prevents. The cheap ones run first and block. The expensive ones are path-filtered and can be declared degraded, on the record, with an expiry. And somebody looks at the configuration every quarter and asks what each stage actually examined last time it ran.

That last habit is the one that keeps the rest honest, and it is the only part of this lesson that no tool will do for you.