Objective
By the end of this lab you will have authored the artefacts that
implement an Ansible CI pipeline: a workflow that runs three
checks in parallel — ansible-lint for static analysis,
ansible-playbook --syntax-check for parsing, and molecule test for integration testing — plus the configuration that
makes each check work, the Molecule scenario that exercises the
role in a disposable container, the failure-mode catalogue, and
a comparison of local vs CI Molecule runs.
The point of this lab is not any single tool — the labs in
Lessons LI-03 through LI-05 covered ansible-lint, syntax
checks, and Molecule respectively. The point is the
integration: three checks in a single workflow, with the
correct severity policy for each (lint errors fail, Molecule
idempotency warnings report), and the scenario files that make
Molecule reproducible across runners.
Architecture
A pipeline with three parallel jobs: lint runs ansible-lint,
syntax runs ansible-playbook --syntax-check, and molecule
runs the full Molecule scenario (create → prepare → converge →
idempotency → verify → destroy). The three jobs are independent;
a failure in one does not block the others.
flowchart LR
A["push / pull_request"] --> L["lint\nansible-lint"]
A --> S["syntax\nansible-playbook --syntax-check"]
A --> M["molecule test"]
L -- fail --> Z1["PR red"]
S -- fail --> Z1
M -- fail --> Z1
L -- pass --> G["merge allowed"]
S -- pass --> G
M -- pass --> G
The three jobs are deliberately independent. A lint failure does not delay Molecule; a Molecule failure does not mask a lint error. The reviewer sees all three results in the PR check list.
Requirements
- Git 2.55.x on Linux or macOS.
- A GitHub repository with the Ansible role under
roles/web/and a top-level playbook atsite.yml. The lab builds both from scratch in Task 1. - Docker for the Molecule Docker driver. The runner must have Docker available; on GitHub-hosted runners it does, but the workflow must declare the dependency.
- No network access required for Ansible Galaxy in CI; the
workflow installs roles from a pinned
requirements.yml.
Scenario
A platform team runs Ansible against a fleet of Linux hosts.
Every role is in a repository, with a Molecule scenario that
exercises the role in a Docker container. The CI pipeline runs
three checks: ansible-lint for code quality, syntax check for
parsing, and molecule test for integration. The team wants all
three to run on every PR; the failure of any one blocks the
merge.
The lab builds the workflow, the lint configuration, the Molecule scenario, and the failure-mode catalogue.
Tasks
Task 1 — Build the role and playbook
LAB="$HOME/ansible-ci-lab"
rm -rf "$LAB"
mkdir -p "$LAB"
cd "$LAB"
git init -b main
git config user.email 'ops@example.com'
git config user.name 'Ops'
mkdir -p roles/web/tasks roles/web/handlers roles/web/defaults
# A minimal web server role that installs nginx and serves a
# static file.
cat > roles/web/tasks/main.yml <<'EOF'
---
- name: install nginx
ansible.builtin.apt:
name: nginx
state: present
update_cache: true
cache_valid_time: 3600
become: true
when: ansible_os_family == 'Debian'
- name: install nginx (RedHat)
ansible.builtin.dnf:
name: nginx
state: present
become: true
when: ansible_os_family == 'RedHat'
- name: deploy index.html
ansible.builtin.copy:
content: |
<html><body>ok</body></html>
dest: /var/www/html/index.html
owner: www-data
group: www-data
mode: '0644'
become: true
notify: restart nginx
- name: ensure nginx is running
ansible.builtin.service:
name: nginx
state: started
enabled: true
become: true
EOF
cat > roles/web/handlers/main.yml <<'EOF'
---
- name: restart nginx
ansible.builtin.service:
name: nginx
state: restarted
become: true
EOF
cat > roles/web/defaults/main.yml <<'EOF'
---
web_listen_port: 80
EOF
# Top-level playbook that uses the role.
cat > site.yml <<'EOF'
---
- name: configure web servers
hosts: webservers
roles:
- role: web
EOF
cat > ansible.cfg <<'EOF'
[defaults]
inventory = inventories/dev/hosts.ini
roles_path = roles
host_key_checking = False
stdout_callback = yaml
forks = 10
EOF
mkdir -p inventories/dev
cat > inventories/dev/hosts.ini <<'EOF'
[webservers]
web1 ansible_connection=local
EOF
git add roles/ site.yml ansible.cfg inventories/
git commit -m 'initial: web role and top-level playbook'
The repository has a web role that installs and configures
nginx, plus a top-level playbook that uses it. The role’s tasks
use FQCN (ansible.builtin.apt, ansible.builtin.copy) so the
lint check can enforce FQCN.
Task 2 — Author the .ansible-lint configuration
# check-shell-blocks: allow-invalid
cd "$HOME/ansible-ci-lab"
cat > .ansible-lint <<'EOF'
---
# ansible-lint 6.x configuration.
profile: production
# ─────────────────────────────────────────────────────────────────
# Rules to skip. The team's triage notes:
# - schema[meta]: noisy on small roles; the team has not yet
# standardised on the schema
# - var-naming[no-role-prefix]: noisy; the team uses unprefixed
# variables in defaults
# - yaml[line-length]: noisy on long URLs in copy modules
# ─────────────────────────────────────────────────────────────────
skip_list:
- schema[meta]
- var-naming[no-role-prefix]
- yaml[line-length]
# ─────────────────────────────────────────────────────────────────
# Rules to enable beyond the production profile.
# - fqcn[action-core]: enforce FQCN for action plugins
# - no-handler: handlers must reference roles, not ad-hoc tasks
# ─────────────────────────────────────────────────────────────────
enable_list:
- fqcn[action-core]
- no-handler
# ─────────────────────────────────────────────────────────────────
# Per-file overrides. The inventories/dev/hosts.ini file is not
# an Ansible file; suppress all findings for it.
# ─────────────────────────────────────────────────────────────────
exclude_paths:
- inventories/
- .github/
- molecule/
# ─────────────────────────────────────────────────────────────────
# The kinds of files ansible-lint will process. .yml, .yaml,
# .j2; jinja templates are processed for `template` references.
# ─────────────────────────────────────────────────────────────────
kinds:
- playbook: "playbooks/*.yml"
- playbook: "site.yml"
- tasks: "roles/*/tasks/*.yml"
- vars: "roles/*/vars/*.yml"
- meta: "roles/*/meta/*.yml"
EOF
git add .ansible-lint
git commit -m 'lint: ansible-lint configuration with FQCN enforcement'
The .ansible-lint file uses the production profile (the
strictest) and adds two rules (fqcn[action-core],
no-handler) that the profile does not include by default.
The skip_list documents three rules the team has triaged as
noisy and intentionally disabled. The exclude_paths block
excludes inventory files, GitHub workflows, and Molecule
scenarios from linting.
Task 3 — Author the Molecule scenario
# check-shell-blocks: allow-invalid
cd "$HOME/ansible-ci-lab"
mkdir -p molecule/default
cat > molecule/default/molecule.yml <<'EOF'
---
driver:
name: docker
platforms:
- name: web1
image: geerlingguy/docker-ubuntu2204-ansible:latest
# The image is a minimal Ubuntu 22.04 with Python and
# systemd. The geerlingguy images are well-known and
# pinned to specific tags by the lab.
privileged: false
command: /lib/systemd/systemd
volumes:
- /sys/fs/cgroup:/sys/fs/cgroup:rw
cgroupns_mode: host
pre_build_image: true
provisioner:
name: ansible
inventory:
group_vars:
all:
ansible_user: ansible
playbooks:
converge: converge.yml
verify: verify.yml
verifier:
name: ansible
scenario:
name: default
test_sequence:
- dependency
- syntax
- create
- prepare
- converge
- idempotence # assert the second converge is a no-op
- verify
- destroy
EOF
cat > molecule/default/converge.yml <<'EOF'
---
- name: converge
hosts: all
gather_facts: true
roles:
- role: web
EOF
cat > molecule/default/verify.yml <<'EOF'
---
- name: verify
hosts: all
gather_facts: true
tasks:
- name: assert nginx is running
ansible.builtin.service:
name: nginx
register: svc
- name: assert index.html is served
ansible.builtin.uri:
url: http://localhost/index.html
status_code: 200
return_content: true
register: page
- name: assert page content
ansible.builtin.assert:
that:
- "'ok' in page.content"
fail_msg: "index.html does not contain 'ok'"
EOF
git add molecule/
git commit -m 'molecule: default scenario with docker driver'
The Molecule scenario uses the Docker driver with a
geerlingguy Ubuntu image. The test_sequence runs seven steps:
dependency (install Galaxy roles), syntax (parse check),
create (provision the container), prepare (run prep
playbook if any), converge (apply the role), idempotence
(re-apply and assert no changes), verify (assert the outcome),
and destroy (tear down the container).
The idempotence step is the critical one: it re-applies the
role and asserts the second run reports zero changes. A role
that changes state on every run is not idempotent and is a
correctness bug, not a test failure.
Task 4 — Author the CI workflow
# check-shell-blocks: allow-invalid
cd "$HOME/ansible-ci-lab"
mkdir -p .github/workflows
cat > .github/workflows/ansible-ci.yml <<'EOF'
name: ansible ci
on:
pull_request:
branches: [main]
push:
branches: [main]
permissions:
contents: read
concurrency:
group: ansible-ci-${ github.ref }
cancel-in-progress: ${ github.ref != 'refs/heads/main' }
jobs:
# ─────────────────────────────────────────────────────────────────
# ansible-lint
# ─────────────────────────────────────────────────────────────────
lint:
name: ansible/lint
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
- name: install ansible-lint
run: |
python3 -m pip install \
--user \
'ansible-lint==6.14.*'
- name: run ansible-lint
run: |
~/.local/bin/ansible-lint \
--exclude .github/ \
--exclude inventories/ \
--exclude molecule/
# ─────────────────────────────────────────────────────────────────
# syntax check
# ─────────────────────────────────────────────────────────────────
syntax:
name: ansible/syntax
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
- name: install ansible-core
run: |
sudo apt-get update
sudo apt-get install -y ansible-core
- name: syntax check
run: |
for pb in site.yml molecule/default/converge.yml molecule/default/verify.yml; do
ansible-playbook --syntax-check "$pb"
done
# ─────────────────────────────────────────────────────────────────
# molecule
# ─────────────────────────────────────────────────────────────────
molecule:
name: ansible/molecule
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
- name: install molecule and docker driver
run: |
python3 -m pip install \
--user \
'molecule==5.*' \
'molecule-plugins[docker]==23.*' \
'ansible-core'
- name: run molecule
run: |
export PATH="$HOME/.local/bin:$PATH"
molecule test \
--driver-name docker \
--scenario-name default
EOF
git add .github/workflows/ansible-ci.yml
git commit -m 'ci: ansible-lint, syntax check, and molecule'
The workflow has three jobs that run in parallel. The lint
job installs ansible-lint 6.14.x and runs it against the
repository. The syntax job installs ansible-core from apt
and runs ansible-playbook --syntax-check against each
playbook. The molecule job installs Molecule with the Docker
driver and runs molecule test.
The three jobs are independent. A lint failure does not stop the Molecule job, and a Molecule failure does not mask a lint error. The reviewer sees all three results.
Task 5 — Document the failure modes
# check-shell-blocks: allow-invalid
cd "$HOME/ansible-ci-lab"
cat > failure-modes.md <<'EOF'
# Failure modes: Ansible CI
This document is the canonical record of the common Ansible CI
failures the team has seen in production. Each section includes
the symptom, the cause, and the fix.
## 1. Lint false positives
**Symptom:** `ansible-lint` flags a finding that the team
considers correct.
**Cause:** The default rule set is strict and includes rules
the team has triaged as too noisy.
**Fix:** Add the rule to `.ansible-lint`'s `skip_list` with a
comment explaining why. Disabling silently is a bug; disabling
with a comment is a decision.
Example:
```yaml
skip_list:
- schema[meta] # team has not standardised on meta schemas
- var-naming[no-role-prefix] # team uses unprefixed variables
2. FQCN rule violation
Symptom: ansible-lint reports fqcn[action-core] for an
action like apt instead of ansible.builtin.apt.
Cause: The role was written for Ansible 2.9 or earlier, when FQCNs were not enforced.
Fix: Replace the short name with the FQCN. The
ansible-lint --fix flag rewrites the playbooks automatically
for many rules; run it on the role and review the diff.
ansible-lint --fix roles/web/
3. Molecule Docker daemon unavailable
Symptom: molecule test fails with
Cannot connect to the Docker daemon at unix:///var/run/docker.sock.
Cause: The runner does not have Docker, or the runner’s
user is not in the docker group.
Fix: On GitHub-hosted runners, Docker is available
out-of-the-box. On self-hosted runners (Lab 13), install
Docker and ensure the runner’s user is in the docker group.
Verify with docker ps from the runner.
4. Idempotence failure
Symptom: Molecule reports Idempotence failed after the
second converge.
Cause: The role changes state on every run. Common causes:
commandorshellmodules whose output is non-deterministicfilemodes that conflict with the OS default (e.g., setting mode0644on a file whose default is already0644triggers a no-op diff; setting mode0600always changes the file)lineinfilewith a regex that matches a different line each time
Fix: Replace command/shell with the appropriate module;
remove redundant file modes; verify lineinfile regexes with
the actual file content.
5. Geerlingguy image pull failure
Symptom: Molecule fails at the create step with
Unable to pull image geerlingguy/docker-ubuntu2204-ansible:latest.
Cause: The runner has no network access to Docker Hub, or the image tag has been deleted.
Fix: Pin the image to a specific tag (not latest) and
mirror the image to the team’s internal registry. The lab
uses :latest for readability; production should pin to a
tag.
6. Galaxy role not found
Symptom: The converge step fails with
ERROR! the role 'web' was not found.
Cause: The role is in roles/ but ansible.cfg does not
include roles_path = roles. Molecule runs from the scenario
directory and looks for roles in the working directory.
Fix: Either add roles_path = ../roles to a
molecule/default/ansible.cfg, or symlink the role into the
scenario directory.
EOF
git add failure-modes.md git commit -m ‘docs: failure modes for ansible CI’
The failure-modes document is the on-call reference. Each
section is the answer to a specific failure; the document is
what the engineer reads first when the CI reddens.
### Task 6 — Compare local vs CI Molecule runs
```bash
# check-shell-blocks: allow-invalid
cd "$HOME/ansible-ci-lab"
cat > local-vs-ci-tradeoffs.md <<'EOF'
# Local vs CI Molecule runs
This document compares running `molecule test` on the
developer's laptop against running it in the GitHub Actions
workflow. Both are necessary; neither is sufficient.
## Local run
The developer runs `molecule test` from the role directory:
cd roles/web molecule test
**Advantages:**
- Fast feedback loop (no runner startup, no checkout, no
artefact upload).
- The developer sees the full output, including the idempotence
step's per-task diff.
- The developer can iterate on the role without committing.
**Disadvantages:**
- The developer's Docker version, Molecule version, and Ansible
version may differ from the CI runner's. A failure that
happens on the runner but not locally (or vice versa) is hard
to debug.
- The developer must install Molecule, the Docker driver, and
any collections locally.
- The developer may skip running Molecule at all, in which case
CI is the only check.
## CI run
The CI runs `molecule test` on a `ubuntu-24.04` runner.
**Advantages:**
- The CI runner is a known environment: Docker version, Molecule
version, and Ansible version are pinned in the workflow.
- The CI result is the canonical record: the reviewer sees the
same output the developer saw.
- The CI runs every commit, so the role's history is auditable.
**Disadvantages:**
- Slower (runner startup, checkout, install, container pull).
- The CI result is a black box: the developer cannot interact
with the container; debugging requires reading the log.
- The CI runner may not match the developer's local
environment exactly.
## The team's discipline
1. **Local runs are required before pushing.** A developer
pushes a PR after `molecule test` has passed locally. CI is
not a substitute for local verification.
2. **CI runs are the canonical record.** The PR's Molecule check
is what the reviewer reads; the local run is what the
developer used to debug.
3. **CI failures that do not reproduce locally are investigated
as infrastructure issues.** A flaky Molecule run is a bug in
the scenario (typically a missing `wait_for` or a missing
retry), not a bug in the role.
## When local and CI disagree
The most common cause of disagreement is the Ansible version.
The CI pins `ansible-core` 2.16; the developer's laptop may
have 2.15 or 2.17. Modules that changed behaviour between
versions (rare, but documented) cause local vs CI divergence.
The mitigation is to pin the developer's local Molecule
environment to match CI:
python3 -m pip install
‘ansible-core==2.16.’
‘molecule==5.’
‘molecule-plugins[docker]==23.*’
A team's onboarding documentation should pin these versions
explicitly. The lab's workflow pins `ansible-lint` to 6.14.x;
local development should match.
EOF
git add local-vs-ci-tradeoffs.md
git commit -m 'docs: local vs CI Molecule tradeoffs'
The local-vs-CI comparison is what the team reads when they ask “do I really need to run Molecule locally?”. The answer is yes, because CI is not a substitute for local verification, but the comparison makes the trade-offs explicit.
Task 7 — Validate the YAML structure
cd "$HOME/ansible-ci-lab"
# Workflow parses and has three jobs.
python3 -c "
import yaml
with open('.github/workflows/ansible-ci.yml') as f:
doc = yaml.safe_load(f)
jobs = doc['jobs']
print('jobs:', list(jobs.keys()))
print('lint steps:', len(jobs['lint']['steps']))
print('molecule runs-on:', jobs['molecule']['runs-on'])
"
# Molecule scenario parses.
python3 -c "
import yaml
with open('molecule/default/molecule.yml') as f:
doc = yaml.safe_load(f)
print('driver:', doc['driver']['name'])
print('platforms:', [p['name'] for p in doc['platforms']])
print('test_sequence:', doc['scenario']['test_sequence'])
"
# .ansible-lint parses.
python3 -c "
import yaml
with open('.ansible-lint') as f:
doc = yaml.safe_load(f)
print('profile:', doc['profile'])
print('skip_list:', doc['skip_list'])
print('enable_list:', doc['enable_list'])
"
Expected output (excerpt):
jobs: ['lint', 'syntax', 'molecule']
lint steps: 3
molecule runs-on: ubuntu-24.04
driver: docker
platforms: ['web1']
test_sequence: ['dependency', 'syntax', 'create', 'prepare',
'converge', 'idempotence', 'verify', 'destroy']
profile: production
skip_list: ['schema[meta]', 'var-naming[no-role-prefix]', 'yaml[line-length]']
enable_list: ['fqcn[action-core]', 'no-handler']
The workflow has three jobs; the Molecule scenario uses the
Docker driver with the seven-step test sequence including
idempotence; the .ansible-lint config uses the production
profile with three skipped rules and two enabled rules.
Task 8 — Capture the deliverables
cd "$HOME/ansible-ci-lab"
cp .github/workflows/ansible-ci.yml "$HOME/ansible-ci.yml"
cp .ansible-lint "$HOME/.ansible-lint"
cp molecule/default/molecule.yml "$HOME/molecule.yml"
cp molecule/default/converge.yml "$HOME/molecule-converge.yml"
cp molecule/default/verify.yml "$HOME/molecule-verify.yml"
cp failure-modes.md "$HOME/failure-modes.md"
cp local-vs-ci-tradeoffs.md "$HOME/local-vs-ci-tradeoffs.md"
ls -l "$HOME"/ansible-ci.yml \
"$HOME"/.ansible-lint \
"$HOME"/molecule.yml \
"$HOME"/molecule-converge.yml \
"$HOME"/molecule-verify.yml \
"$HOME"/failure-modes.md \
"$HOME"/local-vs-ci-tradeoffs.md
The deliverables are the seven files in $HOME, plus the
repository at $HOME/ansible-ci-lab.
Validation
.github/workflows/ansible-ci.ymlparses as valid YAML and has three jobs:lint,syntax,molecule..ansible-lintparses as valid YAML, uses theproductionprofile, and includes bothskip_listandenable_list.molecule/default/molecule.ymlparses as valid YAML, uses the Docker driver, and includesidempotencein the test sequence.molecule/default/converge.ymlandverify.ymlare valid playbooks.- Every
uses:reference in the workflow is a pinned commit SHA.
Expected Outcome
An Ansible CI pipeline that runs three checks on every PR:
ansible-lint for static analysis, ansible-playbook --syntax-check for parsing, and molecule test for integration
testing with idempotence.
$HOME/ansible-ci-lab/
├── .github/workflows/ansible-ci.yml # the workflow
├── .ansible-lint # the lint config
├── molecule/default/
│ ├── molecule.yml # the scenario
│ ├── converge.yml # the role apply
│ └── verify.yml # the assertions
├── roles/web/ # the role
├── site.yml # the top-level playbook
├── failure-modes.md # the failure catalogue
└── local-vs-ci-tradeoffs.md # the comparison
The workflow is the implementation; the configurations are the policy; the failure-modes document is the rationale.
Troubleshooting
ansible-lint reports fqcn[action-core]. The action is
using a short name. Run ansible-lint --fix to auto-rewrite,
or replace manually with the FQCN.
molecule test fails at create. The Docker driver cannot
create the container. Verify docker ps works on the runner;
on GitHub-hosted runners, Docker is available out-of-the-box.
On self-hosted runners, install Docker and add the runner user
to the docker group.
molecule test fails at idempotence. The role changes
state on every run. Read the idempotence log: it shows the
diff between the first and second converge. Common causes are
command/shell modules and file modes that conflict with
OS defaults.
ansible-playbook --syntax-check fails on molecule/default/verify.yml.
The verify playbook has a parse error. Run
ansible-playbook --syntax-check molecule/default/verify.yml
locally to see the error.
The role is not found at converge time. ansible.cfg does
not include roles_path = roles. Add it; Molecule runs from
the scenario directory and looks for roles relative to the
working directory.
Cleanup
LAB="$HOME/ansible-ci-lab"
mv "$LAB"/failure-modes.md "$LAB"/local-vs-ci-tradeoffs.md \
"$HOME"/ 2>/dev/null
mv "$LAB/.github/workflows/ansible-ci.yml" \
"$HOME/ansible-ci.yml" 2>/dev/null
mv "$LAB/.ansible-lint" "$HOME/.ansible-lint" 2>/dev/null
mv "$LAB/molecule/default/molecule.yml" \
"$HOME/molecule.yml" 2>/dev/null
mv "$LAB/molecule/default/converge.yml" \
"$HOME/molecule-converge.yml" 2>/dev/null
mv "$LAB/molecule/default/verify.yml" \
"$HOME/molecule-verify.yml" 2>/dev/null
rm -rf "$LAB"
find "$HOME" -maxdepth 1 -name 'ansible-ci-lab' -print
# expected: (no output)
If you ran Molecule locally during the lab, destroy any leftover containers:
docker ps -a --filter 'label=molecule.identifier' --format '{.ID}' \
| xargs -r docker rm -f
What You Learned
- Three checks, three jobs, parallel.
ansible-lint,ansible-playbook --syntax-check, andmolecule testare independent and run in parallel. A failure in one does not delay the others; the reviewer sees all three results. - FQCN enforcement is non-negotiable. The
fqcn[action-core]rule catches short names that break when collections are upgraded. The team’s discipline: FQCN everywhere, no exceptions. skip_listentries are documentation. Disabling a rule silently is a bug; disabling it with a comment is a decision. The team reviewsskip_listchanges in code review.- Idempotence is the real test.
syntax-checkparses;convergeapplies;idempotenceproves the role is correct. A role that fails idempotence is broken, regardless of what the other checks report. - Molecule is integration testing, not unit testing. The
Docker driver provisions a real container; the scenario
exercises the role end-to-end. The team’s discipline: one
defaultscenario per role, no exceptions. - Local and CI runs are paired, not redundant. Local runs are the developer’s fast feedback loop; CI runs are the canonical record. Both are required.
- Ansible version pinning matters. A developer’s local
ansible-core2.15 vs the CI’s 2.16 is enough to cause divergence. Pin both to the same major version.