Molecule and integration testing — scenarios, drivers, and the verify stage

ansible-playbook --syntax-check proves a playbook parses. --check --diff proves a run would change what the diff shows. Neither proves that, after the change is applied to a real system, the system is in the state the author intended. That is what Molecule proves. Molecule is the canonical integration test framework for Ansible roles and playbooks: it provisions an ephemeral host, converges the role, asserts idempotency, and verifies the resulting system state. A green Molecule run is the strongest static and runtime claim an Ansible CI pipeline can make.

What Molecule is

Molecule is a test harness for Ansible roles and playbooks. It manages the lifecycle of an ephemeral test environment - typically a container or virtual machine - and runs the role against it, then runs verifications. The harness is driven from a directory of scenarios; each scenario is a self-contained test definition that names the driver, the platforms, the provisioner, the verifier, and the sequence of stages.

The canonical CI invocation runs the default scenario end-to-end:

molecule test

This single command runs the full lifecycle. Individual stages can be invoked separately - molecule converge, molecule idempotence, molecule verify, molecule destroy - which is what the CI pipeline usually orchestrates explicitly so each stage is a separate gate with its own log.

flowchart LR
    A["molecule test"] --> B[dependency]
    B --> C[lint]
    C --> D[cleanup]
    D --> E[destroy]
    E --> F[side-effect]
    F --> G[syntax]
    G --> H[create]
    H --> I[prepare]
    I --> J[converge]
    J --> K[idempotence]
    K --> L[verify]
    L --> M[cleanup]
    M --> N[destroy]

Each stage is a separate gate with its own pass/fail criteria, and the lifecycle is the order in which they run.

The default lifecycle

The stages of the default scenario, in order, and what each one proves:

1. dependency - runs ansible-galaxy collection install -r collections/requirements.yml and installs role dependencies declared in meta/main.yml. Without this stage, the converge runs against a different collection set than production.
2. lint - runs ansible-lint and yamllint against the role. Same gates as the CI lint job, scoped to the role under test.
3. cleanup - removes any previous converge artefacts (e.g. the molecule/ state directory).
4. destroy - tears down any previously-created ephemeral hosts from a prior run.
5. side-effect - runs any playbook listed under scenario.side_effects in molecule.yml. Used to create external state the role depends on (e.g. seed an S3 bucket).
6. syntax - runs ansible-playbook --syntax-check against the role’s playbooks. The lesson LI-04 gate, scoped to the role.
7. create - provisions the ephemeral host(s) declared under platforms in molecule.yml.
8. prepare - runs the prepare playbook that brings the host to the pre-converge state (e.g. install base packages).
9. converge - runs the role (or the named playbook) against the ephemeral host. This is the first real run; it should report changes for every task that needs to make one.
10. idempotence - runs the converge a second time. A correctly-written role reports zero changes on this run; the assert fails otherwise.
11. verify - runs the verify step, which is where the system state is asserted. This is where testinfra or ansible.builtin.assert lives.
12. cleanup - removes any artefacts the verify stage produced.
13. destroy - tears down the ephemeral host.

A full molecule test runs all of these in order. A pipeline that wants to fail fast on lint and syntax will run the lint and syntax stages as separate jobs; a pipeline that wants the full integration claim runs molecule test as one job.

Drivers: what Molecule proves depends on what it converges against

The driver determines what kind of ephemeral host the role is tested against. The three common choices:

• delegated - Molecule does not manage a host at all. The converge runs against the local machine or a host the user has provisioned externally. Useful for testing roles that target infrastructure Molecule cannot easily containerise (network devices, cloud instances, hypervisors). The CI runner is the host.
• docker - Molecule provisions Docker containers as ephemeral hosts. Each platforms entry in molecule.yml becomes a container. The converge runs against the container over SSH. Fast, cheap, hermetic. The right default for roles that target Linux.
• podman - same as docker, with rootless containers. Useful when the CI runner cannot run a Docker daemon.
• vagrant - Molecule provisions VMs via Vagrant (VirtualBox, libvirt, VMware). Slower than containers but supports kernel-level and boot-time testing. Used for roles that need a full OS lifecycle.

The choice of driver changes what the test proves. A role tested with the docker driver proves that it converges correctly against a Linux container with the chosen base image; it does not prove it works against RHEL 8 on a VM with SELinux enforcing. A team that supports multiple target distributions should have a matrix of platforms in the platforms: section so each base image is exercised.

The verify stage: where the claim is made

The converge stage runs the role; the verify stage asserts what the host looks like afterwards. Two common verifier implementations:

• testinfra - a Python test framework that connects to the ephemeral host (typically over SSH, using the same connection Molecule used for converge) and runs Python assertions about the system: a file exists, a service is running, a package is installed, a port is listening. The verifier file is verify.yml for Ansible-native asserts or tests/test_*.py for testinfra.
• ansible.builtin.assert - an Ansible-native verifier that runs assertions against the host using Ansible modules and assert. The assertions are written in YAML+Jinja, with no Python required. Easier to read for engineers who do not maintain a Python test toolchain.

The verify step is the strongest claim Molecule makes. Without a verify step, Molecule proves the converge succeeded; with a verify step, Molecule proves the converge produced a specific system state. The production pattern is to assert the post-state in detail: the configuration file exists and has the expected content, the service is enabled and running, the package is installed at the expected version, the user exists with the expected shell.

The CI pipeline pattern

The typical CI pipeline runs Molecule as the runtime gate, after the static gates (lint, syntax-check) have passed. Three patterns are common:

1. molecule test as one job - the simplest. Runs the full lifecycle in one CI job. The log is long; the failure mode is obvious because the stage that failed is in the log.
2. Stage-by-stage jobs - separate CI jobs for molecule lint, molecule syntax, molecule converge, molecule idempotence, molecule verify, molecule destroy. Each job is a separate gate with its own badge. Failure points are immediately visible.
3. Matrix of scenarios - separate jobs for each molecule.yml in the molecule/ directory, allowing different drivers or platform sets per scenario. Used by roles that target multiple distributions.

For most teams, pattern 2 with a single scenario is the right balance between visibility and CI cost. Pattern 1 is acceptable when the role is small and the team is small; pattern 3 is necessary when the role ships to multiple targets.

Production discipline

1. Every role in the repository has at least one Molecule scenario in molecule/ with a non-trivial verify step. A role without a scenario is a role that has not been tested.
2. The verify step asserts the system state, not just the absence of errors. A scenario with verify: [] is not a verified scenario.
3. The driver matches the production target family. Testing a Linux role with the delegated driver against the CI runner tests the runner, not the role.
4. molecule test runs in CI, not just locally. A scenario that runs only on a developer’s laptop is a scenario that has no audit trail.
5. collections/requirements.yml is installed before Molecule converges. Molecule’s dependency stage handles this; skipping the stage means converging against a different collection set than production.
6. Idempotence is asserted by a real second run. A scenario that disables the idempotence stage to make CI faster is a scenario that has stopped testing idempotency.

Cross-course references

• Ansible for Production Sysadmins - Part XXVI (Testing) covers the Molecule scenario format and lifecycle in depth; this lesson is the CI integration of that pattern.
• Ansible for Production Sysadmins - Part XXXVIII (GitCI) is the pipeline that runs Molecule; this lesson is the Molecule-specific stage.
• Ansible for Production Sysadmins - Part XXXVII (RepoArch) covers where molecule/ lives in the repository layout.
• This course, Part L (TerraformCI) - lesson git-cicd-gitops-l-05-terratest-and-integration-tests is the Terraform analogue; both produce the strongest runtime claim available to their tool.

Quiz