Capstone 1: Build the estate

This is the first of four capstone labs. They build one estate, in order, and each depends on the artefacts the previous one produced. Do not start this one intending to skip to lab 3.

Objective

By the end of this lab you will have a repository that a colleague can clone and run against staging without asking you a single question, two inventories whose shape states what a change would touch, a variable layering you can derive rather than guess, per-environment vault identities you have proven are separate, and an OS baseline that reports zero changes on its second run.

The artefact that matters most is not any playbook. It is the blast-radius table. Every later lab refers back to it.

Architecture

Seven machines: one controller and six managed nodes. Production is five hosts across three tiers. Staging is one host carrying all three tiers at once, which is a deliberate and important compromise.

                        ctrl  192.0.2.10
                     Ansible controller
                     git checkout + vault passwords + SSH key
                              │
        ┌─────────────────────┼─────────────────────┐
        │                     │                     │
   inventories/          roles/ + playbooks/     reports/
   production            shared by both env      run evidence
   staging                                       (never committed)
        │
        ├── production ─────────────────────────────────────┐
        │     lb01   192.0.2.11   loadbalancers   ingress   │
        │     app01  192.0.2.21   appservers                │
        │     app02  192.0.2.22   appservers                │
        │     app03  192.0.2.23   appservers                │
        │     db01   192.0.2.31   databases                 │
        │                                                   │
        └── staging ────────────────────────────────────────┤
              stg01  198.51.100.21   loadbalancers          │
                                     appservers             │
                                     databases              │

Five production hosts and one staging host is the budget. A reader with two spare VMs should add lb02 at 192.0.2.12 and db02 at 192.0.2.32; the inventory is shaped so that is a two-line change, and Task 3 asks you to work out what those two lines do to the blast-radius table before you make them.

Requirements

• A controller with ansible-core 2.21.x and Python 3.12+. Everything in these four labs was checked against core 2.21.3.
• Six managed VMs with systemd as PID 1, their own kernel and their own network stack. Modes: B-nested only.

• Debian 12/13 or Ubuntu 24.04 on every node. Pick one family and stay on it across all four labs; the package and service names diverge.
• SSH key access from the controller to every node, as a non-root user with sudo.
• Out-of-band access to every node — hypervisor console or serial. Task 7 replaces the SSH daemon configuration. Name the console procedure for your hypervisor before you start it, not after.
• A VM snapshot of every node taken before Task 1. All four labs assume you can roll back to it.
• Roughly 3.5 hours.

Scenario

You have inherited an estate that is currently administered by hand and by three shell scripts of unknown provenance. There is no repository, no inventory, and the only record of which hosts exist is a wiki page last edited fourteen months ago.

Your first job is not to automate anything. It is to write down what exists, in a form a machine can read, so that every later change can state its blast radius before it runs.

Tasks

Task 1: Capture the starting state

Nothing in this capstone assumes a fresh machine, and Cleanup in lab 4 restores from what you capture here. Do this first, on every node.

WORKDIR="$HOME/estate"
mkdir -p "$WORKDIR/reports/pre-capstone"
cd "$WORKDIR"

Write a throwaway inventory just to reach the hosts. The real one comes in Task 3.

# bootstrap-inventory.yml
all:
  hosts:
    lb01: {ansible_host: 192.0.2.11}
    app01: {ansible_host: 192.0.2.21}
    app02: {ansible_host: 192.0.2.22}
    app03: {ansible_host: 192.0.2.23}
    db01: {ansible_host: 192.0.2.31}
    stg01: {ansible_host: 198.51.100.21}
  vars:
    ansible_user: operator

# capture.yml
- name: Record the pre-capstone state of every node
  hosts: all
  become: true
  gather_facts: true

  tasks:
    - name: Read the sshd effective configuration
      ansible.builtin.command: sshd -T
      register: sshd_effective
      changed_when: false

    - name: Read the installed package list
      ansible.builtin.package_facts:
        manager: auto

    - name: Read enabled units
      ansible.builtin.command: systemctl list-unit-files --state=enabled --no-pager
      register: units
      changed_when: false

    - name: Write the capture to the controller
      ansible.builtin.copy:
        content: |
          host: {{ inventory_hostname }}
          captured: {{ ansible_date_time.iso8601 }}
          distribution: {{ ansible_facts.distribution }} {{ ansible_facts.distribution_version }}
          kernel: {{ ansible_facts.kernel }}
          default_ipv4: {{ ansible_facts.default_ipv4.address | default('none') }}
          package_count: {{ ansible_facts.packages | length }}
          sshd_permitrootlogin: >-
            {{ sshd_effective.stdout_lines
               | select('match', '^permitrootlogin')
               | list | first | default('unknown') }}
          sshd_passwordauthentication: >-
            {{ sshd_effective.stdout_lines
               | select('match', '^passwordauthentication')
               | list | first | default('unknown') }}
          enabled_units: |
            {{ units.stdout | indent(12) }}
        dest: "{{ playbook_dir }}/reports/pre-capstone/{{ inventory_hostname }}.yml"
        mode: '0644'
      delegate_to: localhost
      become: false

$ ansible-playbook -i bootstrap-inventory.yml capture.yml

Task 2: The repository skeleton

cd "$HOME/estate"

mkdir -p inventories/production/group_vars/all
mkdir -p inventories/production/host_vars
mkdir -p inventories/staging/group_vars/all
mkdir -p inventories/staging/host_vars
mkdir -p roles playbooks docs reports
mkdir -p roles/baseline/tasks roles/baseline/handlers
mkdir -p roles/baseline/templates roles/baseline/defaults roles/baseline/meta

The configuration file lives in the repository, not in your home directory, so that everyone who clones it gets the same behaviour.

# ansible.cfg
[defaults]
inventory = inventories/staging/hosts.yml
roles_path = roles
forks = 10
host_key_checking = True
retry_files_enabled = False
stdout_callback = default
display_skipped_hosts = False
vault_identity_list = staging@~/.estate-vault/staging, production@~/.estate-vault/production
vault_id_match = True
interpreter_python = auto
log_path = reports/ansible.log

[privilege_escalation]
become = False
become_method = sudo
become_user = root
become_ask_pass = False

[ssh_connection]
pipelining = True
ssh_args = -o ControlMaster=auto -o ControlPersist=120s -o PreferredAuthentications=publickey

Three of those lines are load-bearing and worth stating plainly.

inventory points at staging. The default target of an accidental ansible-playbook site.yml with no -i should never be production. This single line is the cheapest guardrail in the repository.

vault_id_match = True means a vault-id-labelled file is only tried against the password with the matching label. Without it, Ansible tries every password it holds against every encrypted file, and the separation you build in Task 6 becomes decorative — a controller that holds both passwords would open production while you believed you were in staging.

become = False at the top level means privilege escalation is a decision each play makes, visible in the play, rather than an ambient default that applies to tasks nobody reviewed.

$ ansible-config dump --only-changed
ansible --version | head -3

CONFIG_FILE() = /home/operator/estate/ansible.cfg
DEFAULT_FORKS(/home/operator/estate/ansible.cfg) = 10
DEFAULT_HOST_LIST(/home/operator/estate/ansible.cfg) = ['/home/operator/estate/inventories/staging/hosts.yml']
DEFAULT_LOG_PATH(/home/operator/estate/ansible.cfg) = /home/operator/estate/reports/ansible.log
DEFAULT_VAULT_IDENTITY_LIST(/home/operator/estate/ansible.cfg) = ['staging@~/.estate-vault/staging', 'production@~/.estate-vault/production']

ansible [core 2.21.3]
config file = /home/operator/estate/ansible.cfg

If CONFIG_FILE() is not the path you expect, stop and fix that before anything else. Every later claim in this capstone assumes the repository config is in effect.

Now the ignore file, which is a security control and not housekeeping:

# .gitignore
reports/
*.retry
.vault-*
*.pem
*.key
id_ed25519*
__pycache__/

The estate uses two collections beyond ansible.builtin, and both are pinned. An unpinned collection is a dependency that changes underneath you between the staging run and the production run.

# requirements.yml
collections:
  - name: ansible.posix
    version: '>=1.6.0,<3.0.0'
  - name: community.general
    version: '>=10.0.0,<12.0.0'

$ ansible-galaxy collection install -r requirements.yml -p collections/
ansible-galaxy collection list | head -20

Add collections/ to .gitignore and collections_path = collections to the [defaults] section of ansible.cfg if you want the checkout self-contained; install to the user path instead if your controller is rebuilt from requirements.yml each time. Lab 4 rebuilds the controller and this is one of the decisions it tests.

Write the layout document now, while the reasons are fresh. This is the first deliverable.

docs/repo-layout.md

ansible.cfg            The only config. Committed. Points at staging.
inventories/           One directory per environment. Never merged.
  production/hosts.yml    5 hosts. Owner: platform team.
  production/group_vars/  Non-secret production values.
  production/group_vars/all/vault.yml   Encrypted, id "production".
  staging/...             Same shape. Owner: anyone on the team.
roles/                 Shared by both environments, unconditionally.
                       A role that branches on environment is a bug;
                       the difference belongs in group_vars.
playbooks/             Entry points. Each one names its hosts: line.
docs/                  Blast-radius table, rollback plans, handover.
reports/               Run evidence. GIT-IGNORED. Regenerated, never
                       edited by hand.

Never committed: vault passwords, SSH private keys, anything under
reports/, and any file whose name matches *.pem or *.key.

Task 3: Two inventories, and the blast-radius table

The production inventory. Note that every host appears in exactly one tier group, and the tier groups are collected under estate.

# inventories/production/hosts.yml
all:
  children:
    loadbalancers:
      hosts:
        lb01:
          ansible_host: 192.0.2.11
    appservers:
      hosts:
        app01:
          ansible_host: 192.0.2.21
        app02:
          ansible_host: 192.0.2.22
        app03:
          ansible_host: 192.0.2.23
    databases:
      hosts:
        db01:
          ansible_host: 192.0.2.31
    estate:
      children:
        loadbalancers:
        appservers:
        databases:

The staging inventory puts one host in all three tier groups. That is legal, it is what one machine standing in for three tiers means, and it has a consequence you must record.

# inventories/staging/hosts.yml
all:
  children:
    loadbalancers:
      hosts:
        stg01:
          ansible_host: 198.51.100.21
    appservers:
      hosts:
        stg01:
          ansible_host: 198.51.100.21
    databases:
      hosts:
        stg01:
          ansible_host: 198.51.100.21
    estate:
      children:
        loadbalancers:
        appservers:
        databases:

Print both graphs. This output is a deliverable — save it.

$ ansible-inventory -i inventories/production/hosts.yml --graph \
| tee reports/graph-production.txt

ansible-inventory -i inventories/staging/hosts.yml --graph \
| tee reports/graph-staging.txt

@all:
|--@ungrouped:
|--@loadbalancers:
|  |--lb01
|--@appservers:
|  |--app01
|  |--app02
|  |--app03
|--@databases:
|  |--db01
|--@estate:
|  |--@loadbalancers:
|  |  |--lb01
|  |--@appservers:
|  |  |--app01
|  |  |--app02
|  |  |--app03
|  |--@databases:
|  |  |--db01

Now count, per group, per environment. --list-hosts answers “what would this pattern touch” without connecting to anything, and it is the single habit that separates an operator who knows the blast radius from one who finds out afterwards.

$ for g in loadbalancers appservers databases estate all; do
n=$(ansible -i inventories/production/hosts.yml "$g" --list-hosts \
    | tail -n +2 | grep -c . || true)
printf '%-16s %s\n' "$g" "$n"
done | tee reports/blast-radius-production.txt

loadbalancers    1
appservers       3
databases        1
estate           5
all              5

Write docs/blast-radius.md from those numbers. The table is the deliverable, and the last two columns are the ones that make it useful.

Two rows deserve argument rather than acceptance.

estate and all are marked “never target for a change” not because targeting them is impossible but because a group whose membership grows silently is a group whose blast radius grows silently. all gains every host anyone adds to the inventory, including the one somebody adds at 16:50 on a Friday. Read-only fact gathering against all is fine. A change against all should require somebody to type the tier groups out, because typing them is the moment they think about the count.

The staging column is the uncomfortable one. Staging has one host in every group, so staging cannot prove anything about batching. A serial: 1 rollout across a group of one is a single unbatched run. Lab 3 returns to this; record it now, in the table, as a known limit of your test environment rather than discovering it during a production rollout.

Task 4: Variable layers

Three layers, and each one exists because of a question it answers.

group_vars/all/main.yml answers “what is true of every host in this environment”. group_vars/<tier>.yml answers “what is true of this tier”. host_vars/<host>.yml answers “what is true of this machine only” — and should be nearly empty, because a populated host_vars directory is the shape of a snowflake estate.

# inventories/production/group_vars/all/main.yml
estate_env: production
estate_domain: example.com

baseline_timezone: Etc/UTC
baseline_packages:
  - chrony
  - curl
  - rsync
  - jq

app_name: estate-app
app_version: '1.0.0'
app_port: 8080
app_health_path: /health

db_host: db01
db_name: estate
db_user: estate_app

# inventories/staging/group_vars/all/main.yml
estate_env: staging
estate_domain: staging.example.com

baseline_timezone: Etc/UTC
baseline_packages:
  - chrony
  - curl
  - rsync
  - jq

app_name: estate-app
app_version: '1.0.0'
app_port: 8080
app_health_path: /health

db_host: stg01
db_name: estate
db_user: estate_app

Tier files carry only what the tier needs:

# inventories/production/group_vars/appservers.yml
app_workers: 4
app_log_level: warning

# inventories/staging/group_vars/appservers.yml
app_workers: 1
app_log_level: debug

Now prove it rather than believing it. ansible-inventory --host renders every variable that host would receive from inventory sources, with the layering already resolved.

$ ansible-inventory -i inventories/production/hosts.yml --host app01 \
| tee reports/vars-app01.json

{
  "ansible_host": "192.0.2.22",
  "app_health_path": "/health",
  "app_log_level": "warning",
  "app_name": "estate-app",
  "app_port": 8080,
  "app_version": "1.0.0",
  "app_workers": 4,
  "baseline_timezone": "Etc/UTC",
  "db_host": "db01",
  "db_name": "estate",
  "db_user": "estate_app",
  "estate_domain": "example.com",
  "estate_env": "production"
}

Fill in the variable-layer map — the third deliverable. Pick five variables and name the file each effective value came from:

Task 5: The guardrail that refuses the wrong target

Every playbook in this estate imports one file first. It is short and it has saved more outages than any other twenty lines in a repository.

# playbooks/guard.yml — imported by every entry point, never run alone
- name: Refuse to run a change against production without an explicit limit
  ansible.builtin.assert:
    that:
      - ansible_limit is defined
      - ansible_limit | length > 0
    fail_msg: >-
      This is {{ estate_env }} and no --limit was given, so this play would
      target all {{ ansible_play_hosts_all | length }} hosts in the pattern.
      Re-run with --limit naming the hosts you intend to change, or with
      --limit estate if you genuinely mean all of them.
    success_msg: >-
      {{ estate_env }}: limited to {{ ansible_limit }}
      ({{ ansible_play_hosts_all | length }} host(s) in scope)
  when: estate_env == 'production'
  run_once: true
  delegate_to: localhost
  become: false

ansible_limit is a magic variable holding the contents of --limit. It is undefined when no limit was given, which is exactly the condition worth refusing.

Task 6: Vault identities, one per environment

Two passwords, both outside the repository, both mode 0600.

$ install -d -m 0700 "$HOME/.estate-vault"
umask 077

# Generate, do not invent. Two different passwords.
openssl rand -base64 32 > "$HOME/.estate-vault/staging"
openssl rand -base64 32 > "$HOME/.estate-vault/production"

chmod 0600 "$HOME/.estate-vault/staging" "$HOME/.estate-vault/production"
ls -l "$HOME/.estate-vault"

Create the encrypted variable files. The labels staging and production are written into the file header, which is how vault_id_match decides which password to try.

# inventories/production/group_vars/all/vault.yml — before encryption
vault_db_password: REPLACE_ME_PRODUCTION
vault_app_secret: REPLACE_ME_PRODUCTION

# inventories/staging/group_vars/all/vault.yml — before encryption
vault_db_password: REPLACE_ME_STAGING
vault_app_secret: REPLACE_ME_STAGING

$ ansible-vault encrypt \
--vault-id production@"$HOME/.estate-vault/production" \
inventories/production/group_vars/all/vault.yml

ansible-vault encrypt \
--vault-id staging@"$HOME/.estate-vault/staging" \
inventories/staging/group_vars/all/vault.yml

head -1 inventories/production/group_vars/all/vault.yml
head -1 inventories/staging/group_vars/all/vault.yml

$ANSIBLE_VAULT;1.2;AES256;production
$ANSIBLE_VAULT;1.2;AES256;staging

The label is in the header, in clear text. That is by design: it tells the tool which key to reach for without telling anybody what the secret is.

Now prove the separation. This is the deliverable, and it is the step most people skip.

$ ansible-vault view \
--vault-id staging@"$HOME/.estate-vault/staging" \
inventories/production/group_vars/all/vault.yml

Decryption failed (no vault secrets were found that could decrypt).
Origin: .../inventories/production/group_vars/all/vault.yml

That failure is the evidence. Save the command and its output into reports/vault-separation.txt. A claim that staging and production secrets are separate, unaccompanied by a failed decryption, is an assertion about a configuration file rather than a fact about the estate.

Task 7: The access model

One automation account, key-based, with a narrow and auditable sudo grant. Do this through Ansible so it is reproducible, and do it while you still have a working login as operator.

# roles/baseline/tasks/access.yml
- name: Create the automation account
  ansible.builtin.user:
    name: "{{ automation_user }}"
    shell: /bin/bash
    create_home: true
    state: present

- name: Install the controller's public key
  ansible.posix.authorized_key:
    user: "{{ automation_user }}"
    key: "{{ lookup('file', automation_pubkey_path) }}"
    state: present
    exclusive: true

- name: Grant a narrow sudo rule, validated before it is installed
  ansible.builtin.copy:
    dest: "/etc/sudoers.d/60-{{ automation_user }}"
    content: |
      # Managed by Ansible. Local edits are overwritten.
      {{ automation_user }} ALL=(root) NOPASSWD: ALL
    mode: '0440'
    owner: root
    group: root
    validate: '/usr/sbin/visudo -cf %s'

- name: Harden sshd with a drop-in, validated before it is installed
  ansible.builtin.copy:
    dest: /etc/ssh/sshd_config.d/60-estate.conf
    content: |
      # Managed by Ansible. Local edits are overwritten.
      PermitRootLogin no
      PasswordAuthentication no
      KbdInteractiveAuthentication no
      PubkeyAuthentication yes
    mode: '0644'
    owner: root
    group: root
    validate: '/usr/sbin/sshd -t -f %s'
  notify: Reload sshd

# roles/baseline/handlers/main.yml
- name: Reload sshd
  ansible.builtin.systemd_service:
    name: ssh
    state: reloaded

Task 8: The baseline role, with an argument spec

# roles/baseline/defaults/main.yml
automation_user: ansible
automation_pubkey_path: ~/.ssh/id_ed25519.pub
baseline_timezone: Etc/UTC
baseline_packages: []
baseline_motd_owner: platform-team

# roles/baseline/meta/argument_specs.yml
argument_specs:
  main:
    short_description: OS baseline for every host in the estate
    options:
      automation_user:
        type: str
        required: false
        default: ansible
        description: Unprivileged account the controller connects as.
      automation_pubkey_path:
        type: path
        required: true
        description: Path on the controller to the public key to install.
      baseline_timezone:
        type: str
        required: false
        default: Etc/UTC
      baseline_packages:
        type: list
        elements: str
        required: false
        default: []
      baseline_motd_owner:
        type: str
        required: true
        description: Team named in the managed banner. No default on purpose.

An argument spec turns “the role did something strange because a variable was misspelled” into “the role refused to start and named the variable”. It runs before any task, so a role that would have half-applied fails having applied nothing.

# roles/baseline/tasks/main.yml
- name: Access model
  ansible.builtin.import_tasks: access.yml

- name: Install the baseline packages
  ansible.builtin.package:
    name: "{{ baseline_packages }}"
    state: present
  when: baseline_packages | length > 0

- name: Set the timezone
  ansible.builtin.command: "timedatectl set-timezone {{ baseline_timezone }}"
  register: tz
  changed_when: tz.rc == 0 and current_tz.stdout | trim != baseline_timezone

- name: Managed banner naming the owner and the repository
  ansible.builtin.template:
    src: motd.j2
    dest: /etc/motd
    mode: '0644'

That timedatectl task is deliberately wrong, and Task 9 is where you notice. Read it again before continuing: it references current_tz, which no task registers, and it runs timedatectl unconditionally on every run.

The honest version reads the current value first and only acts when it differs:

# roles/baseline/tasks/main.yml — the corrected timezone tasks
- name: Read the current timezone
  ansible.builtin.command: timedatectl show --property=Timezone --value
  register: current_tz
  changed_when: false

- name: Set the timezone when it differs
  ansible.builtin.command: "timedatectl set-timezone {{ baseline_timezone }}"
  when: current_tz.stdout | trim != baseline_timezone
  changed_when: true

{# roles/baseline/templates/motd.j2 #}
{{ '#' }} ------------------------------------------------------------
{{ '#' }} {{ inventory_hostname }} — {{ estate_env | upper }}
{{ '#' }} Managed by Ansible. Owner: {{ baseline_motd_owner }}
{{ '#' }} Local changes to managed files are overwritten on the next run.
{{ '#' }} Repository: git@git.example.com:platform/estate.git
{{ '#' }} ------------------------------------------------------------

And the entry point:

# playbooks/baseline.yml
- name: Apply the OS baseline
  hosts: estate
  become: true
  gather_facts: true

  pre_tasks:
    - name: Guardrail
      ansible.builtin.import_tasks: guard.yml

  roles:
    - role: baseline
      baseline_motd_owner: platform-team

Task 9: Run it, then run it again

Staging first. Always staging first — and note that on a one-host staging environment this proves the tasks work, not that the batching works.

$ ansible-playbook -i inventories/staging/hosts.yml \
playbooks/baseline.yml --check --diff

$ ansible-playbook -i inventories/staging/hosts.yml playbooks/baseline.yml

Before touching production, confirm from a second terminal that you can still log in to stg01 as operator and still sudo. If you cannot, the console you opened in Task 7 is why the Requirements asked for it.

Then production, one host at a time to begin with:

$ ansible-playbook -i inventories/production/hosts.yml \
playbooks/baseline.yml --limit app01

Confirm login and sudo on app01, then the rest:

$ ansible-playbook -i inventories/production/hosts.yml \
playbooks/baseline.yml --limit 'estate:!app01'

Now the test that decides whether any of this was real. Run the whole thing a second time, unchanged.

$ ansible-playbook -i inventories/production/hosts.yml \
playbooks/baseline.yml --limit estate | tee reports/baseline-second-run.txt

grep -E 'changed=[1-9]' reports/baseline-second-run.txt || echo 'IDEMPOTENT'

PLAY RECAP *********************************************************************
app01   : ok=9  changed=0  unreachable=0  failed=0  skipped=1  rescued=0  ignored=0
app02   : ok=9  changed=0  unreachable=0  failed=0  skipped=1  rescued=0  ignored=0
app03   : ok=9  changed=0  unreachable=0  failed=0  skipped=1  rescued=0  ignored=0
db01    : ok=9  changed=0  unreachable=0  failed=0  skipped=1  rescued=0  ignored=0
lb01    : ok=9  changed=0  unreachable=0  failed=0  skipped=1  rescued=0  ignored=0

IDEMPOTENT

If the uncorrected timedatectl task is still in place, this run reports changed=1 on every host, forever, and the changed_when expression raises an undefined-variable error the moment it evaluates. Fix it with the corrected pair above and run again until the recap is clean. Keep reports/baseline-second-run.txt — it is the deliverable.

Validation

Run each of these and keep the output.

$ cd "$HOME/estate"

# 1. The repository config is the one in effect.
ansible-config dump --only-changed | grep CONFIG_FILE

# 2. Both inventories parse and the counts match the table.
ansible-inventory -i inventories/production/hosts.yml --graph
ansible-inventory -i inventories/staging/hosts.yml --graph

# 3. Effective variables are derivable for a named host.
ansible-inventory -i inventories/production/hosts.yml --host app01

# 4. The staging password cannot open production. Must fail.
ansible-vault view --vault-id staging@"$HOME/.estate-vault/staging" \
inventories/production/group_vars/all/vault.yml || echo 'SEPARATION OK'

# 5. The guardrail refuses an unlimited production run.
ansible-playbook -i inventories/production/hosts.yml \
playbooks/baseline.yml --check || echo 'GUARDRAIL OK'

# 6. Second-run idempotency.
grep -E 'changed=[1-9]' reports/baseline-second-run.txt || echo 'IDEMPOTENT'

# 7. No secret material in the tree.
grep -rIl 'ANSIBLE_VAULT' inventories/ | sort
git check-ignore -v reports/ansible.log

Every line must pass:

• CONFIG_FILE() names the repository’s ansible.cfg, not ~/.ansible.cfg.
• Production graphs 5 hosts across 3 tier groups; staging graphs 1 host in 3 tier groups.
• app01 resolves app_workers: 4 from the tier file and estate_env: production from all.
• The cross-environment decrypt fails, and you have the failure saved.
• The unlimited production run fails at the guardrail assertion, naming the host count.
• The second baseline run reports changed=0 on all five production hosts.
• Exactly two files under inventories/ carry a vault header, and reports/ is git-ignored.

Expected Outcome

estate/
├── ansible.cfg
├── .gitignore
├── bootstrap-inventory.yml
├── capture.yml
├── docs/
│   ├── blast-radius.md
│   ├── repo-layout.md
│   └── variable-layers.md
├── inventories/
│   ├── production/
│   │   ├── hosts.yml
│   │   ├── group_vars/all/main.yml
│   │   ├── group_vars/all/vault.yml     (encrypted, id "production")
│   │   └── group_vars/appservers.yml
│   └── staging/
│       ├── hosts.yml
│       ├── group_vars/all/main.yml
│       ├── group_vars/all/vault.yml     (encrypted, id "staging")
│       └── group_vars/appservers.yml
├── playbooks/
│   ├── baseline.yml
│   └── guard.yml
├── roles/baseline/
│   ├── defaults/main.yml
│   ├── handlers/main.yml
│   ├── meta/argument_specs.yml
│   ├── tasks/{main.yml,access.yml}
│   └── templates/motd.j2
└── reports/                              (git-ignored)
    ├── ansible.log
    ├── baseline-second-run.txt
    ├── blast-radius-production.txt
    ├── graph-{production,staging}.txt
    ├── pre-capstone/*.yml
    ├── vars-app01.json
    └── vault-separation.txt

Six hosts carrying an identical baseline, an automation account with key access and sudo, two encrypted variable files that cannot open each other, and a documented blast radius for every group in both environments.

Troubleshooting

ansible-config dump reports a different CONFIG_FILE. You are not in the repository root, or the directory is world-writable and Ansible skipped it. ls -ld . — if the mode ends in 7, fix it with chmod o-w ..

Ansible prompts for a vault password despite vault_identity_list. Either the config file was not loaded (see above) or the ~ in the path was not expanded. Ansible expands ~ in that setting, but a path written as $HOME/... in the ini file is not shell-expanded — ini files do no variable substitution. Use ~.

Decryption failed when you expected success. Check the header with head -1. If the label does not match any id in vault_identity_list, and vault_id_match = True, no password is tried at all.

The argument spec fails with “missing required arguments”. It names the variable. baseline_motd_owner has no default on purpose; supply it in the play, not in defaults/, so that a new caller is forced to think about who owns the host.

visudo -cf rejects the sudoers file. Read the message: it names the line. The commonest cause is a Jinja expression that rendered empty, so the line begins with a space and ALL=(root) with no user in front. Render the template to a scratch file and read it.

sshd -t -f passes but you still cannot log in. The syntax was valid; the policy was wrong. PasswordAuthentication no locks out anyone without a key, including you if your key is not in ~/.ssh/authorized_keys for the account you use. Recover on the console, add the key, and re-run.

authorized_key with exclusive: true removed a key you needed. That is what exclusive means: the module makes the file contain exactly the keys you specified. It is the right setting for an automation account whose access should be knowable, and the wrong setting for a shared human account. Recover from reports/pre-capstone/ or the console.

UNREACHABLE on one host, exit code 4. The host is in the inventory and not answering. Confirm with ansible -i inventories/production/hosts.yml appservers -m ping, and remember the recap distinguishes unreachable from failed for a reason: an unreachable host ran no tasks at all, so it is in whatever state it was in before the play started.

Cleanup

This lab is the foundation of the next three. Do not run Cleanup unless you are abandoning the capstone. If you are continuing to lab 2, skip this section entirely.

If you are stopping here, restore in reverse order. The sshd drop-in comes off first, because removing the automation account while sshd still refuses password authentication can leave you with no way in.

Step 1. Restore SSH access to what the capture recorded:

$ cd "$HOME/estate"

ansible -i inventories/production/hosts.yml estate -b \
-m file -a 'path=/etc/ssh/sshd_config.d/60-estate.conf state=absent'

ansible -i inventories/production/hosts.yml estate -b \
-m systemd_service -a 'name=ssh state=reloaded'

ansible -i inventories/production/hosts.yml estate -b \
-m command -a 'sshd -T' | grep -E 'passwordauthentication|permitrootlogin'

Compare that output against reports/pre-capstone/*.yml. If the values differ from the capture, something else on the host is setting them and you need to know what before you call this restored.

Step 2. Remove the sudo grant and the automation account:

$ ansible -i inventories/production/hosts.yml estate -b \
-m file -a 'path=/etc/sudoers.d/60-ansible state=absent'

ansible -i inventories/production/hosts.yml estate -b \
-m user -a 'name=ansible state=absent remove=yes'

Step 3. Restore the message of the day. /etc/motd was overwritten by the template; on a Debian-family default install it is empty, and the capture recorded whether yours was:

$ ansible -i inventories/production/hosts.yml estate -b \
-m copy -a 'content="" dest=/etc/motd mode=0644'

Step 4. Remove the vault passwords and the checkout from the controller. Do this last, and only when you are certain you do not want the evidence:

$ shred -u "$HOME/.estate-vault/staging" "$HOME/.estate-vault/production"
rmdir "$HOME/.estate-vault"

mkdir -p "$HOME/estate-deliverables"
cp -a "$HOME/estate/docs" "$HOME/estate/reports" "$HOME/estate-deliverables/"

rm -rf "$HOME/estate"

Step 5. Roll the VM snapshots back if you want the nodes truly as they were. The steps above restore configuration; only the snapshot restores the package list.

What You Learned

• The inventory is the blast-radius map. A group is not a label, it is a count, and --list-hosts turns “I think that is three hosts” into a number before anything runs.
• Staging that shares one host across three tiers cannot prove batching. Recording that as a known limit is worth more than pretending the environments match.
• Variable layers must be derivable. ansible-inventory --host resolves inventory sources, and knowing that it does not show role defaults, set_fact or --extra-vars is as important as reading it.
• vault_id_match = True is what makes vault identities a separation rather than a naming convention, and a failed cross-environment decrypt is the only acceptable proof.
• validate: runs the daemon’s own parser before the file is installed, which converts the two classic lockouts — a broken sudoers, a broken sshd_config — into a task failure with the file untouched.
• The second run is the test. A task that reports changed every time poisons the recap, fires handlers nobody asked for, and makes check mode lie.