The Ansible CI discipline — what "tested" means for Ansible

The word “tested” on an Ansible pull request is doing more work than it looks. A change that has been “tested” can mean the YAML parses, the playbook parses, the static rules are clean, the dry run reports no changes, and a Molecule scenario converged and verified against an ephemeral container. These are five different claims about five different properties of the change. Conflating them is how untested playbooks reach production with the badge “CI passed”.

Static checks versus runtime checks

The gates available to an Ansible pipeline split cleanly into two families. Static checks read the files in the working tree and report what they find without ever connecting to a managed host. Runtime checks actually invoke the playbook against a target (ephemeral or otherwise) and observe what it would do or what it did.

flowchart LR
    A[Working tree YAML] --> B[Static checks]
    B --> B1["yamllint"]
    B --> B2["ansible-lint"]
    B --> B3["ansible-playbook --syntax-check"]
    B1 --> C[Runtime checks]
    B2 --> C
    B3 --> C
    C --> C1["ansible-playbook --check --diff"]
    C --> C2["molecule converge"]
    C --> C3["molecule verify"]

Static checks answer “is this code well-formed?”. Runtime checks answer “does this code do what the author intended, against a real system?”. Both questions must be answered; neither is a substitute for the other.

What each layer proves

Each gate in the pipeline is scoped to a specific claim. The CI run is the union of those claims:

A “CI passed” badge that summarises all five is a much stronger claim than a badge that summarises only the first three. The discipline of this Part is to make all five explicit and to make the absence of any one of them visible.

The boundary between CI and apply

The apply step is where Ansible stops reading files and starts changing systems. Everything in this Part sits on one side of that boundary; the apply sits on the other side. The point of the CI pipeline is to make the boundary explicit: when CI passes, the change is well-formed, lints clean, parses, plans to do something defensible, and converges on an ephemeral host to a verified state. The apply on a production fleet then becomes a routine execution of an already-validated change, not a leap of faith.

A team that runs only static checks and skips the runtime gates is treating the boundary as “files look plausible”. A team that runs runtime checks on ephemeral hosts but skips the apply on protected environments is treating the boundary as “it works in Molecule” - which is true but is a claim about disposable containers, not production. The discipline is to be specific about which claim is being made at each layer.

Production discipline

1. All five gates run on every pull request that touches Ansible files. Skipping a gate for “small changes” is how the gate that catches the small change gets removed.
2. CI badges name the gates they cover. A green badge that aggregates lint + syntax-check is a different claim from a green badge that aggregates lint + syntax-check + molecule. Be specific in the pipeline status names.
3. A Molecule scenario that has not been run in CI is a scenario that has not been tested. Local Molecule runs do not produce an audit trail and do not gate the merge.
4. The apply job is the only job allowed to mutate production hosts. Every gate before it is read-only or runs against ephemeral infrastructure.
5. Treat unverified Molecule scenarios as untested code. A scenario file in the repository is documentation, not evidence.

Cross-course references

• Ansible for Production Sysadmins - Part X (YAML) covers the YAML rules that yamllint enforces.
• Ansible for Production Sysadmins - Part XXV (CheckDiff) covers the --check and --diff semantics that this lesson layers into the pipeline.
• Ansible for Production Sysadmins - Part XXVI (Testing) covers the Molecule scenario format that lesson LI-05 instantiates.
• Ansible for Production Sysadmins - Part XXXVIII (GitCI) is the canonical Ansible CI pipeline; this Part is the GitOps-course instantiation of that pattern.

Quiz