Ansible final assessment - audit an inherited production repository

You have joined Meridian Retail as the engineer responsible for configuration management. The person who built the automation left three weeks ago. You have been handed a repository, a handover note, and forty production hosts that it manages today.

Nobody has told you anything is wrong with it. As far as the business is concerned this repository works: it deploys, the pipeline is green, and the last incident anyone connects to it was months ago.

Your job is to find out whether that is true, and to say so in a form somebody can act on.

Format

• Part A — 19 auto-scored questions, in the set above this text. Machine-marked, 25% of the total. Several of them cannot be answered without having read the repository properly.
• Part B — the defect register you write yourself. 45% of the total. The marking key is in this page, under a heading that says not to read it yet.
• Part C — 6 scenarios with rubrics, in the frontmatter of this assessment and summarised near the end. 30% of the total.
• Open-book, 3 hours. You may run any read-only Ansible command you like against a copy of the repository.
• Pass: 75% overall, plus all three mandatory findings. The pass bar is justified where the pass criteria are set out.

The handover note

The previous engineer left this in HANDOVER.md on the default branch. It is reproduced exactly.

# Handover - Ansible estate

Deploy:      ansible-playbook playbooks/deploy-app.yml
Staging:     ansible-playbook -i inventories/staging/hosts.yml \
               playbooks/deploy-app.yml --limit staging
Rollback:    re-run with -e app_version=<previous version>.
             Ansible is idempotent so re-running is always safe.

Notes for whoever picks this up:

- Staging inventory is a copy of production with the names changed.
  Use --limit staging for anything that is not a full deploy.
- web07 and web08 have been unreachable since the January move. I
  added ignore_unreachable to the pre-flight so the pipeline would
  go green again. Somebody should work out what happened to them.
- meridian_ops was forked from a Galaxy namespace that got pulled
  after a maintainer account takeover. We host our own tarball now
  so we are not affected.
- vault.yml still needs encrypting. OPS-2291, opened in March,
  nobody assigned.
- Do not run the deploy on a Friday.

The repository as you received it

meridian-ops/
├── ansible.cfg
├── requirements.yml
├── HANDOVER.md
├── .gitlab-ci.yml
├── inventories/
│   ├── production/
│   │   ├── hosts.yml
│   │   └── group_vars/
│   │       ├── all.yml
│   │       └── vault.yml
│   └── staging/
│       ├── hosts.yml
│       └── group_vars/
│           └── all.yml
├── vars/
│   └── common.yml
├── playbooks/
│   ├── preflight.yml
│   └── deploy-app.yml
└── roles/
    ├── common/
    │   └── tasks/main.yml
    └── webapp/
        ├── tasks/main.yml
        ├── handlers/main.yml
        └── templates/app.conf.j2

ansible.cfg

[defaults]
inventory = inventories/production/hosts.yml
host_key_checking = False
forks = 50
deprecation_warnings = False
retry_files_enabled = False
roles_path = roles

[ssh_connection]
ssh_args = -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o ControlPersist=60s
pipelining = True

[galaxy]
server_list = internal_galaxy
ignore_certs = True

inventories/production/hosts.yml

all:
  children:
    webservers:
      hosts:
        web[01:12].example.com:
    appservers:
      hosts:
        app[01:22].example.com:
    dbservers:
      hosts:
        db[01:02].example.com:
    loadbalancers:
      hosts:
        lb[01:02].example.com:
    monitoring:
      hosts:
        mon[01:02].example.com:

inventories/staging/hosts.yml

all:
  children:
    staging:
      children:
        webservers:
        appservers:
        canary:
    webservers:
      hosts:
        stage-web[01:02].example.com:
    appservers:
      hosts:
        stage-app01.example.com:
    dbservers:
      hosts:
        stage-db01.example.com:
    # Added 2026-02-14 so the new dashboards could be exercised
    # against real traffic shapes.
    canary:
      hosts:
        web01.example.com:
        web02.example.com:
        app01.example.com:

inventories/production/group_vars/all.yml

app_port: 8080
app_tls_enabled: true
tls_cert_dir: /etc/pki/app

inventories/production/group_vars/vault.yml

# This is the live production credential set. Encrypt this file with
# ansible-vault before go-live. OPS-2291.
vault_db_password: REPLACE_ME_prod_appdb
vault_monitoring_api_key: REPLACE_ME_mon_api_key
db_password: "{{ vault_db_password }}"

inventories/staging/group_vars/all.yml

app_environment: staging
app_port: 8080
app_tls_enabled: "false"
db_host: stage-db01.example.com

vars/common.yml

app_environment: production
app_version: latest
db_host: db01.example.com
db_port: 5432
app_listen_address: 0.0.0.0

playbooks/preflight.yml

- name: Pre-flight
  hosts: all
  gather_facts: false
  tasks:
    - name: Confirm every host answers
      ansible.builtin.ping:
      ignore_unreachable: true

playbooks/deploy-app.yml

- name: Deploy the application
  hosts: all
  become: true
  gather_facts: false
  vars_files:
    - ../vars/common.yml
  roles:
    - common
    - webapp

roles/common/tasks/main.yml

- name: Ensure the deploy user exists
  ansible.builtin.user:
    name: deploy
    groups: sudo
    append: true

- name: Allow the application port through the firewall
  ansible.builtin.shell: iptables -A INPUT -p tcp --dport {{ app_port }} -j ACCEPT

- name: Ensure required packages are present
  ansible.builtin.package:
    name: "{{ item }}"
    state: latest
  with_items:
    - curl
    - rsync
    - jq

roles/webapp/tasks/main.yml

- name: Stop the application before the upgrade
  ansible.builtin.systemd:
    name: appd
    state: stopped

- name: Fetch and unpack the release
  ansible.builtin.shell: |
    cd /opt/app
    curl -sSL http://artifacts.example.com/app/{{ app_version }}.tar.gz -o /tmp/app.tgz
    tar xzf /tmp/app.tgz --strip-components=1
    ./scripts/install.sh --force

- name: Apply database migrations
  ansible.builtin.command: /opt/app/bin/migrate --apply
  ignore_errors: true

- name: Write the application configuration
  ansible.builtin.template:
    src: app.conf.j2
    dest: /etc/app/app.conf
    owner: root
    mode: '0644'
  notify: restart appd

- name: Record the deployment
  ansible.builtin.shell: echo "{{ app_version }} deployed" >> /var/log/deploy.log

roles/webapp/handlers/main.yml

- name: restart appd
  ansible.builtin.systemd:
    name: appd
    state: restarted
    daemon_reload: true

roles/webapp/templates/app.conf.j2

[server]
listen = {{ app_listen_address }}:{{ app_port }}
workers = {{ app_workers | default(4) }}
environment = {{ app_environment }}

[database]
host = {{ db_host }}
port = {{ db_port }}
user = appuser
password = {{ db_pasword | default('') }}

{% if app_tls_enabled %}
[tls]
certificate = {{ tls_cert_dir }}/{{ inventory_hostname }}.pem
private_key = {{ tls_cert_dir }}/{{ inventory_hostname }}.key
{% endif %}

requirements.yml

collections:
  - name: community.general
  - name: ansible.posix
  - name: community.crypto
  - name: https://storage.example.net/ansible/meridian_ops-1.0.0.tar.gz
    type: url

roles:
  - src: https://github.com/example-org/ansible-role-appdeploy
    scm: git
    version: main

.gitlab-ci.yml

stages: [validate, deploy]

variables:
  ANSIBLE_FORCE_COLOR: "1"
  APP_API_TOKEN: "REPLACE_ME_meridian_api_token"

validate:
  stage: validate
  script:
    - ansible-galaxy install -r requirements.yml --force
    - ansible-playbook playbooks/deploy-app.yml --syntax-check
    - ansible-lint playbooks/ roles/ || true

deploy:
  stage: deploy
  only:
    - main
  script:
    - ansible-galaxy install -r requirements.yml --force
    - ansible-playbook playbooks/preflight.yml
    - ansible-playbook playbooks/deploy-app.yml --diff -v -e app_tls_enabled=false ; rc=$?
    - if [ "$rc" -eq 2 ]; then echo "deployment failed" ; exit 1 ; fi
    - echo "deployment complete"

What you must produce

Six deliverables. They are marked together, and an answer that produces findings without severities, or remediation without validation, is incomplete rather than partially correct.

1. Assessment. A defect register. One row per finding: where it is, what it is, and what it can do. Findings that only exist in combination get their own row, and say which components they need.
2. Evidence. For each finding, the read-only command or the specific artefact that establishes it. “It looks wrong” is not evidence. --list-hosts output is.
3. Risk rating. A severity per finding, from the scale below, with one sentence of reasoning. The reasoning is what is marked; the label on its own is not.
4. Remediation. What you change, and in what order. The order is marked. A plan that fixes the most interesting defect first has chosen wrong.
5. Validation. For each remediation, the command whose output demonstrates it worked. Not “the file was updated” — the rendered template, the resolved host list, the recap.
6. Production recommendation. May this repository keep running against the fleet? Under what conditions? What is the blocking set, and what is merely on the backlog?

The severity scale

Use this scale, and use all of it.

Marking key: the defect register

Twenty-nine entries, in two tables. The twenty-seven below are the substantive findings; the two after them are judgement calls that a complete register still has to make.

Score two points for each finding you produced independently, and one further point where your severity is within one band of the key and your reasoning holds.

And two items that must appear in the register with the right verdict:

The interactions

Findings in isolation understate this repository. Six combinations matter more than their parts, and the register is expected to name them.

The fleet-wide outage machine (F8 + F13 + F14 + F17). Production is the default target; the play has no batching; forks exceeds the host count; one handler restarts everything it is notified on. Any single one of these is a code-review comment. Together they convert a one-line template mistake into forty hosts down at once, with no healthy population to compare against and nothing to roll back from.

The staging illusion (F2 + F3). The inventory says three production hosts are in staging. Variable precedence says the staging inventory renders production values anyway. So a “staging” run writes production database details onto production hosts, and the engineer who is being careful — the one who remembered --limit staging — is the one it happens to.

The silent breakage (F9 + F10 + F11 + F12). Two template defects produce a file that renders cleanly and is wrong. No validation runs before the service is restarted, and no health check runs after. The deployment reports changed and ok for a host that is now serving errors, and the first person to know will be a customer.

The false green (F4 + F19 + F20 + F27 + F6). ignore_errors converts a failure into an ignore. ignore_unreachable converts an unreachable host into an ok. The exit-code test misses the code that both of those produce together. The one gate that does object — the linter — has its exit status thrown away by || true. And the job that reports all this success prints the production database password into its log while doing it. Five independent mechanisms, each of which converts a signal into silence, arranged in series.

The undetermined estate (F15 + F16 + F22). Nothing that executes as root on these hosts is pinned, and neither the dependency transport nor the artefact transport authenticates its source. The repository cannot state what ran during the last deployment, which means it also cannot reproduce it, which means every incident investigation starts from a guess.

The commitment with no exit (F5 + F21). The play stops the service in its first task, so it has committed to finishing before it has done anything reversible. The handover note claims a rollback that does not exist. The combination is what produces the 02:40 incident in scenario three: the run failed midway, the operator reached for the documented rollback, and the documented rollback was a sentence somebody wrote once.

Verified evidence

The following outputs were produced against this exact repository on ansible-core 2.21.3, Python 3.14, Jinja 3.1.6. They are what your own evidence column should look like.

ansible-playbook playbooks/deploy-app.yml --list-hosts

$ ansible-playbook playbooks/deploy-app.yml --list-hosts

playbook: playbooks/deploy-app.yml

play #1 (all): Deploy the application	TAGS: []
  pattern: ['all']
  hosts (40):
    app01.example.com
    web12.example.com
    app13.example.com
    ...

$ ansible-inventory -i inventories/staging/hosts.yml --graph

@all:
|--@ungrouped:
|--@staging:
|  |--@webservers:
|  |  |--stage-web01.example.com
|  |  |--stage-web02.example.com
|  |--@appservers:
|  |  |--stage-app01.example.com
|  |--@canary:
|  |  |--web01.example.com
|  |  |--web02.example.com
|  |  |--app01.example.com
|--@dbservers:
|  |--stage-db01.example.com

Three of the six hosts under staging are production. This is the finding that the whole audit turns on, and it took one read-only command and no privileges to establish.

$ ansible-playbook -i inventories/staging/hosts.yml vars-probe.yml

# with vars_files: ../vars/common.yml
ok: [stage-web01.example.com] =>
  "msg": "env=production db_host=db01.example.com tls=false (str)"
ok: [web01.example.com] =>
  "msg": "env=production db_host=db01.example.com tls=false (str)"

# with the vars_files line removed
ok: [stage-web01.example.com] =>
  "msg": "env=staging db_host=stage-db01.example.com tls=false (str)"
ok: [web01.example.com] =>
  "msg": "env=staging db_host=stage-db01.example.com tls=false (str)"

The staging inventory sets db_host: stage-db01.example.com. With vars_files present, every host in that inventory renders db01.example.com instead. The directory layout claims two environments; the precedence order says there is one.

$ ansible-playbook render-probe.yml --limit web01.example.com -e app_tls_enabled=false

[server]
listen = 0.0.0.0:8080
workers = 4
environment = production

[database]
host = db01.example.com
port = 5432
user = appuser
password =

[tls]
certificate = /etc/pki/app/web01.example.com.pem
private_key = /etc/pki/app/web01.example.com.key

Two defects in one file. password = is empty, because db_pasword | default('') turned a misspelling into a silent empty string. And the TLS block was written despite -e app_tls_enabled=false, because a non-empty string is truthy in Jinja.

$ ansible-playbook -i inventories/production/hosts.yml playbooks/preflight.yml

fatal: [web07.example.com]: UNREACHABLE! => ... ignoring

PLAY RECAP
web01.example.com  : ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
web07.example.com  : ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=1

ok=1 and unreachable=0 for a host that does not answer. Only ignored=1 separates it from a host that genuinely converged, and nobody reads that column. Without ignore_unreachable the same run reports unreachable=1 and exits 4.

$ ansible-playbook -i inventory.yml serial-probe.yml

RUNNING HANDLER [restart appd]
  "msg": "RESTART h01"
  "msg": "RESTART h02"
  "msg": "RESTART h03"

RUNNING HANDLER [restart appd]
  "msg": "RESTART h04"
  "msg": "RESTART h05"
  "msg": "RESTART h06"

RUNNING HANDLER [restart appd]
  "msg": "RESTART h07"
  "msg": "RESTART h08"

This is why serial is the fix for the handler problem rather than a mitigation of it. Handlers flush at the end of each batch, so batching the play batches the restarts. Without serial there is one batch, and the handler restarts every host in it.

$ ansible-playbook playbooks/deploy-app.yml --syntax-check

playbook: playbooks/deploy-app.yml

One line, exit 0. Every finding in the register survives this gate, and the pipeline reports it as validation.

The other gate is more interesting, because it does object — and the job discards the objection.

$ ansible-lint --offline playbooks/ roles/

# Rule Violation Summary

1 command-instead-of-shell profile:basic tags:command-shell,idiom
1 name                     profile:basic tags:idiom
1 package-latest           profile:basic tags:idempotency
1 ignore-errors            profile:basic tags:unpredictability
4 no-changed-when          profile:basic tags:command-shell,idempotency

Failed: 8 failure(s), 0 warning(s) in 7 files processed of 9 encountered.

Read that summary against the register. The linter finds F4, F22, F23 and F24, plus a lowercase handler name and the non-idempotent log append in the last task of the webapp role — real findings, correctly raised, and every one of them in the shell and idempotency family. It says nothing about the production hosts in the staging group, nothing about variable precedence, nothing about either template defect, nothing about forty simultaneous restarts, and nothing about the credentials in group_vars.

So the honest conclusion is not “linting is theatre”. It is that linting catches a specific and useful class of defect, that this repository has silenced even that, and that the defects which would end the company are in the class no linter can reach.

Scenarios

Six scenarios, with full rubrics in this assessment’s frontmatter. Items marked REQUIRED are pass or fail on their own.

Pass criteria

• 75% overall: Part A auto-scored at 25%, the defect register at 45%, the scenarios at 30%.
• All three mandatory findings present in Part B, each with a defensible severity:
  • F1, the plaintext production credentials.
  • F2, the production hosts inside the staging group.
  • F4, ignore_errors between a schema migration and a service restart.
• At least three of the six interactions named, because a register of isolated items misrepresents the risk even when every item is correct.
• At least one finding rated Low or None with reasoning. An audit that cannot say what is not urgent has not made a judgement.
• Four of six scenarios passed against their rubrics, with no REQUIRED element missed in any scenario answer.

Why the bar is 75 and not 70

The rest of this course passes coursework at 70. This assessment is set higher deliberately, for two reasons.

The first is the marking scheme. Part B awards marks for findings that a careless reader still stumbles into — the plaintext password is not hard to see, and neither is ignore_errors: true. A candidate who skims the repository, reports the six most visible items, and writes nothing about severity or interaction can assemble something close to 70% out of the obvious half. The bar has to sit above that, or the result certifies the wrong thing.

The second is what the credential claims. Everything in this repository runs as root on forty hosts carrying customer traffic. The question this assessment answers is not “does this person know Ansible” — it is “would I let this person sign off the automation”. That is a higher bar in real life and it should be a higher bar here.

The mandatory findings exist for the same reason. A candidate could reach 75% on breadth while missing the wrong inventory group, and a repository that ships with that defect deploys production changes under the belief that it is staging. Breadth does not compensate for that, so the scoring does not let it.