Capstone 2: Deploy the application

Capstone lab 2 of 4. It continues in the repository built in capstone lab 1 and assumes the baseline, the two inventories and the vault identities already exist.

Objective

By the end you will have a working three-tier service on the estate, built by three roles that know nothing about which environment they are in, with every configuration file validated by its own daemon before it is activated, and a health gate you have proven can fail — because a health check that has never failed is a health check nobody has tested.

Architecture

The service the estate exists to run. Traffic enters at the proxy, is distributed across three application hosts, and every application host talks to one database.

             client
               │  http://192.0.2.11/
               ▼
        ┌─────────────┐
        │    lb01     │  haproxy, backend estate_backend
        │  192.0.2.11 │  admin socket /run/haproxy/admin.sock
        └──────┬──────┘  check GET /health every 2s, fall 2, rise 2
               │
     ┌─────────┼─────────┐
     ▼         ▼         ▼
  app01     app02     app03      estate-app.service, :8080
  .21       .22       .23        GET /health -> 200 or 503
     └─────────┼─────────┘
               ▼
            ┌──────┐
            │ db01 │  postgresql, database "estate"
            │ .31  │  role estate_app, password from vault
            └──────┘

On staging all four boxes are stg01. The topology is identical, the addresses collapse, and that difference lives entirely in inventories/staging/, not in any role.

Requirements

• Everything from capstone lab 1, completed, including the baseline applied and the second run reporting changed=0.
• ansible-core 2.21.x, plus community.general for the haproxy module and ansible.posix — both pinned in requirements.yml from lab 1.
• Six VMs with systemd as PID 1. B-nested only. This lab installs systemd units, reloads the daemon, and restarts services; a container without a real init cannot host any of it, and lab 4’s reboots make the requirement permanent for the whole capstone.
• PostgreSQL available in the distribution repositories (Debian 12/13, Ubuntu 24.04 all ship it).
• Port 8080 free on the application hosts and port 80 free on lb01.
• Roughly 3.5 hours.

Scenario

The baseline is applied and the estate is legible, and now it has to actually run something. The application is a small HTTP service that reports its version and whether it can reach the database. That is a deliberately modest application, because the interesting part of this lab is not the software — it is that every file it depends on is rendered, validated, and only then activated, and that the play can tell whether the result works.

Tasks

Task 1: Capture what these three hosts do today

cd "$HOME/estate"
mkdir -p reports/pre-deploy

# playbooks/capture-services.yml
- name: Record the service state of the estate before deployment
  hosts: estate
  become: true
  gather_facts: true

  tasks:
    - name: Read listening sockets
      ansible.builtin.command: ss -lntp
      register: sockets
      changed_when: false

    - name: Check for an existing haproxy configuration
      ansible.builtin.stat:
        path: /etc/haproxy/haproxy.cfg
      register: haproxy_cfg

    - name: Preserve an existing haproxy configuration
      ansible.builtin.copy:
        src: /etc/haproxy/haproxy.cfg
        dest: /etc/haproxy/haproxy.cfg.pre-capstone
        remote_src: true
        mode: '0644'
        force: false
      when: haproxy_cfg.stat.exists

    - name: Check for an existing PostgreSQL cluster
      ansible.builtin.stat:
        path: /var/lib/postgresql
      register: pgdata

    - name: Write the capture
      ansible.builtin.copy:
        content: |
          host: {{ inventory_hostname }}
          captured: {{ ansible_date_time.iso8601 }}
          haproxy_cfg_existed: {{ haproxy_cfg.stat.exists }}
          postgresql_datadir_existed: {{ pgdata.stat.exists }}
          listening: |
            {{ sockets.stdout | indent(12) }}
        dest: "{{ playbook_dir }}/../reports/pre-deploy/{{ inventory_hostname }}.yml"
        mode: '0644'
      delegate_to: localhost
      become: false

$ ansible-playbook -i inventories/production/hosts.yml \
playbooks/capture-services.yml --limit estate

force: false on that backup copy is the important detail. Re-running this play must not overwrite a good pre-capstone backup with the configuration the capstone itself installed — which is exactly what force: true, the default, would do on the second run.

Task 2: The role interface

Three roles, and one rule that decides whether the design holds: no role may branch on estate_env.

A role that contains when: estate_env == 'production' has moved an environment decision from the inventory, where it is visible in one file and reviewable, into the role, where it is scattered across tasks and applies to every future environment nobody has thought about yet.

The difference between staging and production in this estate is entirely these values:

# inventories/production/group_vars/appservers.yml
app_workers: 4
app_log_level: warning
app_health_timeout_seconds: 2

# inventories/staging/group_vars/appservers.yml
app_workers: 1
app_log_level: debug
app_health_timeout_seconds: 5

# inventories/production/group_vars/databases.yml
db_listen_addresses: '192.0.2.31'
db_max_connections: 100
db_allowed_cidr: '192.0.2.0/24'

# inventories/staging/group_vars/databases.yml
db_listen_addresses: '127.0.0.1'
db_max_connections: 20
db_allowed_cidr: '127.0.0.1/32'

# inventories/production/group_vars/loadbalancers.yml
lb_bind_address: '192.0.2.11'
lb_backend_check_interval: 2s
lb_backend_fall: 2
lb_backend_rise: 2

# inventories/staging/group_vars/loadbalancers.yml
lb_bind_address: '127.0.0.1'
lb_backend_check_interval: 2s
lb_backend_fall: 2
lb_backend_rise: 2

Task 3: The database role

mkdir -p roles/estate_db/{tasks,handlers,templates,defaults,meta}

# roles/estate_db/defaults/main.yml
db_name: estate
db_user: estate_app
db_listen_addresses: '127.0.0.1'
db_max_connections: 20
db_allowed_cidr: '127.0.0.1/32'
db_port: 5432

# roles/estate_db/meta/argument_specs.yml
argument_specs:
  main:
    short_description: PostgreSQL for the estate application
    options:
      db_name:
        type: str
        required: true
      db_user:
        type: str
        required: true
      db_password:
        type: str
        required: true
        description: Supplied from the environment vault. Never defaulted.
      db_listen_addresses:
        type: str
        required: true
        description: Addresses the cluster binds. A default here would be a security default.
      db_allowed_cidr:
        type: str
        required: true
      db_max_connections:
        type: int
        required: false
        default: 20

db_password and db_listen_addresses are required: true with no default, on purpose. A default listen address is a security decision made by whoever wrote the role for every future caller; a default password is worse. Forcing the caller to supply both means the value is in the inventory, where it is reviewable.

# roles/estate_db/tasks/main.yml
- name: Install PostgreSQL
  ansible.builtin.package:
    name:
      - postgresql
      - python3-psycopg2
    state: present

- name: Ensure the cluster is running before configuring it
  ansible.builtin.systemd_service:
    name: postgresql
    state: started
    enabled: true

- name: Discover the cluster configuration directory
  ansible.builtin.shell: |
    set -o pipefail
    ls -1d /etc/postgresql/*/main 2>/dev/null | sort -V | tail -1
  args:
    executable: /bin/bash
  register: pgconf
  changed_when: false

- name: Refuse to continue if no cluster configuration was found
  ansible.builtin.assert:
    that: pgconf.stdout | trim | length > 0
    fail_msg: >-
      No /etc/postgresql/*/main directory on {{ inventory_hostname }}.
      The package installed but the cluster was not created, so there is
      nothing to configure. Check `pg_lsclusters`.

- name: Render the estate configuration drop-in
  ansible.builtin.template:
    src: estate.conf.j2
    dest: "{{ pgconf.stdout | trim }}/conf.d/estate.conf"
    owner: postgres
    group: postgres
    mode: '0640'
    backup: true
    validate: '/usr/lib/postgresql/postgresql-check-conf %s'
  notify: Restart postgresql

# roles/estate_db/tasks/main.yml — the corrected rendering and validation
- name: Render the estate configuration drop-in
  ansible.builtin.template:
    src: estate.conf.j2
    dest: "{{ pgconf.stdout | trim }}/conf.d/estate.conf"
    owner: postgres
    group: postgres
    mode: '0640'
    backup: true
  register: pgdropin
  notify: Restart postgresql

- name: Apply the configuration now, so the assertion below tests the new file
  ansible.builtin.meta: flush_handlers

- name: Confirm the cluster came back and is accepting connections
  ansible.builtin.command: "pg_isready -h {{ db_listen_addresses }} -p {{ db_port }}"
  register: pgready
  changed_when: false
  retries: 10
  delay: 3
  until: pgready.rc == 0

- name: Confirm the cluster is listening where the configuration said
  ansible.builtin.wait_for:
    host: "{{ db_listen_addresses }}"
    port: "{{ db_port }}"
    state: started
    timeout: 30

{# roles/estate_db/templates/estate.conf.j2 #}
{{ '#' }} Managed by Ansible — roles/estate_db. Local edits are overwritten.
listen_addresses = '{{ db_listen_addresses }}'
port = {{ db_port }}
max_connections = {{ db_max_connections }}
log_line_prefix = '%m [%p] %q%u@%d '
log_min_duration_statement = 500

Now the database and role themselves. There is no guaranteed collection here, so these use command with an explicit idempotency check — which is the pattern worth learning anyway, because most estates have at least one tool with no module.

# roles/estate_db/tasks/main.yml — append after the tasks above
- name: Check whether the application role exists
  ansible.builtin.command: >-
    psql -tAc "SELECT 1 FROM pg_roles WHERE rolname = '{{ db_user }}'"
  become_user: postgres
  register: role_exists
  changed_when: false

- name: Create the application role with the vaulted password
  ansible.builtin.command:
    argv:
      - psql
      - -v
      - ON_ERROR_STOP=1
      - -c
      - >-
        CREATE ROLE {{ db_user }} LOGIN PASSWORD '{{ db_password }}'
  become_user: postgres
  when: role_exists.stdout | trim != '1'
  changed_when: true
  no_log: true

- name: Check whether the database exists
  ansible.builtin.command: >-
    psql -tAc "SELECT 1 FROM pg_database WHERE datname = '{{ db_name }}'"
  become_user: postgres
  register: db_exists
  changed_when: false

- name: Create the database
  ansible.builtin.command: "createdb -O {{ db_user }} {{ db_name }}"
  become_user: postgres
  when: db_exists.stdout | trim != '1'
  changed_when: true

# roles/estate_db/handlers/main.yml
- name: Restart postgresql
  ansible.builtin.systemd_service:
    name: postgresql
    state: restarted

Task 4: The application role and a health check that can fail

The application is a single Python file. What matters about it is the /health endpoint: it returns 200 only when it can reach the database, and 503 otherwise.

mkdir -p roles/estate_app/{tasks,handlers,templates,defaults,meta,files}

{# roles/estate_app/templates/app.py.j2 #}
{{ '#' }}!/usr/bin/env python3
{{ '#' }} Managed by Ansible — roles/estate_app. Local edits are overwritten.
import json, os, socket
from http.server import BaseHTTPRequestHandler, HTTPServer

VERSION = "{{ app_version }}"
DB_HOST = "{{ db_host_address }}"
DB_PORT = {{ db_port }}
TIMEOUT = {{ app_health_timeout_seconds }}

def db_reachable():
    try:
        with socket.create_connection((DB_HOST, DB_PORT), TIMEOUT):
            return True
    except OSError:
        return False

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == "{{ app_health_path }}":
            ok = db_reachable() and not os.path.exists("/etc/estate-app-fail")
            body = json.dumps({
                "version": VERSION,
                "host": socket.gethostname(),
                "database": "reachable" if db_reachable() else "unreachable",
                "status": "ok" if ok else "unfit",
            }).encode()
            self.send_response(200 if ok else 503)
        else:
            body = json.dumps({"version": VERSION,
                               "host": socket.gethostname()}).encode()
            self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, fmt, *args):
        pass

if __name__ == "__main__":
    HTTPServer(("0.0.0.0", {{ app_port }}), Handler).serve_forever()

The /etc/estate-app-fail marker is the injection point. Lab 3 uses it to break one host mid-rollout; here it is what lets you prove the health gate works at all.

# roles/estate_app/templates/estate-app.service.j2
[Unit]
Description=Estate application {{ app_version }}
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=estate
Group=estate
EnvironmentFile=-/etc/default/estate-app
ExecStart=/usr/bin/python3 /opt/estate-app/app.py
Restart=on-failure
RestartSec=2
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/log/estate-app

[Install]
WantedBy=multi-user.target

# roles/estate_app/defaults/main.yml
app_name: estate-app
app_version: '1.0.0'
app_port: 8080
app_health_path: /health
app_health_timeout_seconds: 2
app_workers: 1
app_log_level: info

# roles/estate_app/tasks/main.yml
- name: Create the service account
  ansible.builtin.user:
    name: estate
    system: true
    shell: /usr/sbin/nologin
    create_home: false

- name: Create the directories the unit needs
  ansible.builtin.file:
    path: "{{ item }}"
    state: directory
    owner: estate
    group: estate
    mode: '0755'
  loop:
    - /opt/estate-app
    - /var/log/estate-app

- name: Resolve the database address from the inventory
  ansible.builtin.set_fact:
    db_host_address: "{{ hostvars[db_host].ansible_host | default(db_host) }}"

- name: Render the application, validated as Python before it is activated
  ansible.builtin.template:
    src: app.py.j2
    dest: /opt/estate-app/app.py
    owner: root
    group: estate
    mode: '0644'
    backup: true
    validate: 'python3 -m py_compile %s'
  notify: Restart estate-app

- name: Render the unit file
  ansible.builtin.template:
    src: estate-app.service.j2
    dest: /etc/systemd/system/estate-app.service
    owner: root
    group: root
    mode: '0644'
    backup: true
  notify:
    - Reload systemd
    - Restart estate-app

- name: Write the environment file
  ansible.builtin.copy:
    content: |
      APP_VERSION={{ app_version }}
      APP_WORKERS={{ app_workers }}
      APP_LOG_LEVEL={{ app_log_level }}
    dest: /etc/default/estate-app
    owner: root
    group: root
    mode: '0644'
  notify: Restart estate-app

- name: Enable and start the service
  ansible.builtin.systemd_service:
    name: estate-app
    state: started
    enabled: true
    daemon_reload: true

# roles/estate_app/handlers/main.yml
- name: Reload systemd
  ansible.builtin.systemd_service:
    daemon_reload: true

- name: Restart estate-app
  ansible.builtin.systemd_service:
    name: estate-app
    state: restarted

Task 5: The load balancer role

mkdir -p roles/estate_lb/{tasks,handlers,templates,defaults,meta}

{# roles/estate_lb/templates/haproxy.cfg.j2 #}
{{ '#' }} Managed by Ansible — roles/estate_lb. Local edits are overwritten.
global
    log /dev/log local0
    stats socket {{ lb_admin_socket }} mode 660 level admin
    stats timeout 30s
    user haproxy
    group haproxy
    daemon

defaults
    log     global
    mode    http
    option  httplog
    option  dontlognull
    timeout connect 5s
    timeout client  30s
    timeout server  30s

frontend estate_frontend
    bind {{ lb_bind_address }}:80
    default_backend {{ lb_backend_name }}

backend {{ lb_backend_name }}
    balance roundrobin
    option httpchk GET {{ app_health_path }}
    http-check expect status 200
{% for h in groups['appservers'] %}
    server {{ h }} {{ hostvars[h].ansible_host | default(h) }}:{{ app_port }} check inter {{ lb_backend_check_interval }} fall {{ lb_backend_fall }} rise {{ lb_backend_rise }}
{% endfor %}

The for loop over groups['appservers'] is why the load balancer configuration never needs editing when a host is added. It is also why the blast-radius table from lab 1 matters: adding a host to appservers silently changes this file on the next run.

# roles/estate_lb/defaults/main.yml
lb_bind_address: '127.0.0.1'
lb_backend_name: estate_backend
lb_admin_socket: /run/haproxy/admin.sock
lb_backend_check_interval: 2s
lb_backend_fall: 2
lb_backend_rise: 2

# roles/estate_lb/tasks/main.yml
- name: Install haproxy and the socket client the drain step needs
  ansible.builtin.package:
    name:
      - haproxy
      - socat
    state: present

- name: Render the proxy configuration, validated before it is activated
  ansible.builtin.template:
    src: haproxy.cfg.j2
    dest: /etc/haproxy/haproxy.cfg
    owner: root
    group: root
    mode: '0644'
    backup: true
    validate: 'haproxy -c -f %s'
  notify: Reload haproxy

- name: Enable and start the proxy
  ansible.builtin.systemd_service:
    name: haproxy
    state: started
    enabled: true

# roles/estate_lb/handlers/main.yml
- name: Reload haproxy
  ansible.builtin.systemd_service:
    name: haproxy
    state: reloaded

Task 6: Compose the site playbook

# playbooks/site.yml
- name: Database tier
  hosts: databases
  become: true
  gather_facts: true
  pre_tasks:
    - name: Guardrail
      ansible.builtin.import_tasks: guard.yml
  roles:
    - role: estate_db
      db_password: "{{ vault_db_password }}"

- name: Application tier
  hosts: appservers
  become: true
  gather_facts: true
  pre_tasks:
    - name: Guardrail
      ansible.builtin.import_tasks: guard.yml
  roles:
    - role: estate_app

- name: Load balancer tier
  hosts: loadbalancers
  become: true
  gather_facts: true
  pre_tasks:
    - name: Guardrail
      ansible.builtin.import_tasks: guard.yml
  roles:
    - role: estate_lb

Three plays, in dependency order, and the order is not stylistic.

The application’s /health returns 503 until the database is reachable. Deploy the application first and every health gate fails, correctly, for a reason that has nothing to do with the application.

The proxy’s httpchk marks a backend down until /health returns 200. Deploy the proxy first and it starts with every backend down, which on a real estate means the proxy is briefly serving 503 to users while the application tier is still being built.

Write that reasoning into docs/deploy-order.md. The order is obvious once you have seen it fail and invisible before.

Task 7: Deploy staging, then production

$ cd "$HOME/estate"
ansible-playbook -i inventories/staging/hosts.yml \
playbooks/site.yml --check --diff

Check mode will report failures on this first run, and they are not bugs. pg_isready runs against a cluster whose configuration was never written; the estate-app service does not exist so systemd_service has nothing to start. Check mode cannot predict the state of a system it has not yet built. Read the failures, confirm each one is a consequence of the system not existing yet, and move on.

$ ansible-playbook -i inventories/staging/hosts.yml playbooks/site.yml

Prove it works before going anywhere near production:

$ # Substitute your own address if staging is not on loopback:
STG=127.0.0.1

curl -sS "http://$STG:8080/health" | jq .
curl -sS -o /dev/null -w 'proxy status: %{http_code}\n' "http://$STG/"

{
"version": "1.0.0",
"host": "stg01",
"database": "reachable",
"status": "ok"
}
proxy status: 200

Then production, tier by tier, so that a failure stops at a tier boundary rather than half way through the estate:

$ ansible-playbook -i inventories/production/hosts.yml \
playbooks/site.yml --limit databases

$ ansible-playbook -i inventories/production/hosts.yml \
playbooks/site.yml --limit appservers

$ ansible-playbook -i inventories/production/hosts.yml \
playbooks/site.yml --limit loadbalancers

Task 8: The health-gate playbook, and proving it fails

A health check nobody has watched fail is a check nobody has tested. Build the gate as its own playbook so lab 3 can reuse it.

# playbooks/health.yml
- name: Assert the estate is serving
  hosts: appservers
  gather_facts: false
  become: false

  vars:
    health_retries: 10
    health_delay: 3

  tasks:
    - name: Each application host answers 200 on its own address
      ansible.builtin.uri:
        url: "http://{{ ansible_host }}:{{ app_port }}{{ app_health_path }}"
        status_code: 200
        return_content: true
        timeout: 5
      register: health
      retries: "{{ health_retries }}"
      delay: "{{ health_delay }}"
      until: health.status == 200
      delegate_to: localhost

    - name: The host reports the version we deployed
      ansible.builtin.assert:
        that: (health.content | from_json).version == app_version
        fail_msg: >-
          {{ inventory_hostname }} is serving version
          {{ (health.content | from_json).version }}, not {{ app_version }}.
          The file was written but the service is running the old code.
        success_msg: "{{ inventory_hostname }}: {{ app_version }}, database reachable"

- name: Assert the proxy is serving
  hosts: loadbalancers
  gather_facts: false
  become: true

  tasks:
    - name: The proxy answers through the front end
      ansible.builtin.uri:
        url: "http://{{ lb_bind_address }}/"
        status_code: 200
        timeout: 5
      register: proxy_health
      retries: 10
      delay: 3
      until: proxy_health.status == 200
      delegate_to: "{{ inventory_hostname }}"

    - name: Every backend is up
      ansible.builtin.shell: |
        set -o pipefail
        echo "show stat" | socat stdio "{{ lb_admin_socket }}" \
          | awk -F, '$2 !~ /^(BACKEND|FRONTEND)$/ && $1 == "{{ lb_backend_name }}" {print $2, $18}'
      args:
        executable: /bin/bash
      register: backends
      changed_when: false

    - name: Refuse to call the deployment healthy with a backend down
      ansible.builtin.assert:
        that: backends.stdout_lines | reject('search', 'UP') | list | length == 0
        fail_msg: >-
          Backends not UP: {{ backends.stdout_lines
            | reject('search', 'UP') | list | join('; ') }}
        success_msg: >-
          All {{ backends.stdout_lines | length }} backends UP

Run it clean first, so you know what a pass looks like:

$ ansible-playbook -i inventories/production/hosts.yml \
playbooks/health.yml --limit estate

Now break it on purpose. Stop the database and watch the gate refuse.

$ ansible -i inventories/production/hosts.yml databases -b \
-m systemd_service -a 'name=postgresql state=stopped'

$ ansible-playbook -i inventories/production/hosts.yml \
playbooks/health.yml --limit estate \
| tee reports/health-gate-failure.txt

echo "playbook exit status: ${PIPESTATUS[0]}"

TASK [Each application host answers 200 on its own address] *********************
FAILED - RETRYING: [app01]: Each application host answers 200 (10 retries left).
...
fatal: [app01]: FAILED! => {"attempts": 10, "status": 503, ...}
fatal: [app02]: FAILED! => {"attempts": 10, "status": 503, ...}
fatal: [app03]: FAILED! => {"attempts": 10, "status": 503, ...}

PLAY RECAP *********************************************************************
app01  : ok=0  changed=0  unreachable=0  failed=1
app02  : ok=0  changed=0  unreachable=0  failed=1
app03  : ok=0  changed=0  unreachable=0  failed=1

playbook exit status: 2

Keep that file. It is the deliverable, and it is the only evidence that distinguishes a health check from a decorative task that always passes.

Restore the database and confirm the gate goes green again:

$ ansible -i inventories/production/hosts.yml databases -b \
-m systemd_service -a 'name=postgresql state=started'

ansible-playbook -i inventories/production/hosts.yml \
playbooks/health.yml --limit estate

Now write docs/health-checks.md, and make the third section the honest one:

docs/health-checks.md

WHAT EACH CHECK ASSERTS
  uri /health == 200       The process is listening AND can open a TCP
                           connection to the database AND no operator
                           has placed the /etc/estate-app-fail marker.
  version == app_version   The running process is serving the code we
                           deployed, not a stale process from before.
  show stat, all UP        The proxy's own view: it has completed
                           `rise` successful checks against each server.

WHAT A FAILURE MEANS
  503 on one host          That host is unfit. The others may be fine.
  503 on every host        Almost certainly the shared dependency: the
                           database. Check db01 before the app hosts.
  200 but wrong version    The file was written and the service was not
                           restarted. Look for a handler that did not run
                           because an earlier task in the play failed.
  200 direct, backend DOWN The proxy cannot reach the host even though
                           you can. Firewall, bind address, or the proxy
                           is checking a different port.

WHAT THESE CHECKS DO NOT COVER
  - Correctness. The endpoint proves reachability, not that a query
    returns the right answer.
  - Capacity. One successful request says nothing about behaviour at
    load; a host can pass the gate and fall over under real traffic.
  - Data integrity. Nothing here would notice a corrupted table.
  - The database's own health. `db_reachable()` opens a TCP socket. A
    PostgreSQL that is listening but refusing connections because it is
    out of connection slots passes this check.

That last section is why the document exists. A health gate that is believed to cover more than it does is more dangerous than no gate, because it converts “we did not check” into “we checked and it was fine”.

Validation

$ cd "$HOME/estate"

# 1. All three roles refuse to branch on the environment.
grep -rn "estate_env" roles/ || echo 'NO ENVIRONMENT BRANCHING IN ROLES'

# 2. The health gate passes on a healthy estate.
ansible-playbook -i inventories/production/hosts.yml \
playbooks/health.yml --limit estate

# 3. The proxy serves, and reports three backends UP.
ansible -i inventories/production/hosts.yml loadbalancers -b -m shell \
-a 'echo "show stat" | socat stdio /run/haproxy/admin.sock | cut -d, -f1,2,18 | grep estate_backend'

# 4. Every application host serves the deployed version.
ansible -i inventories/production/hosts.yml appservers -m uri \
-a 'url=http://127.0.0.1:8080/health return_content=yes' | grep -o '"version": "[^"]*"'

# 5. Config files carry the managed header and a backup exists.
ansible -i inventories/production/hosts.yml loadbalancers -b -m shell \
-a 'head -1 /etc/haproxy/haproxy.cfg; ls /etc/haproxy/haproxy.cfg.*'

# 6. The failure evidence exists.
grep -c 'failed=1' reports/health-gate-failure.txt

Every line must pass:

• No role mentions estate_env. The environment delta lives in group_vars only.
• health.yml completes with failed=0 on all four hosts in scope.
• show stat reports estate_backend with app01, app02 and app03 all UP.
• All three application hosts serve "version": "1.0.0".
• /etc/haproxy/haproxy.cfg starts with the managed header and a timestamped backup exists beside it.
• reports/health-gate-failure.txt records three failed hosts and exit status 2.

Expected Outcome

estate/
├── docs/
│   ├── deploy-order.md
│   └── health-checks.md
├── playbooks/
│   ├── capture-services.yml
│   ├── health.yml
│   └── site.yml
├── roles/
│   ├── estate_db/{defaults,handlers,meta,tasks,templates}
│   ├── estate_app/{defaults,handlers,meta,tasks,templates}
│   └── estate_lb/{defaults,handlers,meta,tasks,templates}
└── reports/
    ├── health-gate-failure.txt
    └── pre-deploy/*.yml

curl http://192.0.2.11/ returns 200 with a JSON body naming one of three hosts, chosen round-robin. curl http://192.0.2.11/health — via the proxy — returns 200 only while at least one backend is fit. Stopping PostgreSQL takes every backend DOWN within four seconds and the proxy begins returning 503, which is the correct behaviour and the reason the next lab exists.

Troubleshooting

template fails with “failed to validate”. Read the validator’s own message; it is included in the task result. For haproxy -c -f the commonest cause is a server line whose address rendered empty, which means hostvars[h].ansible_host was undefined for one host in appservers.

validate fails with “command not found”. The validator runs on the managed node. Install the package before the template task, not after.

pg_isready times out but PostgreSQL is running. The cluster is listening somewhere else. ss -lntp | grep postgres on db01 shows where. If the file says 192.0.2.31 and the socket says 127.0.0.1, the cluster was reloaded rather than restarted — listen_addresses needs a restart.

/health returns 503 and database reads unreachable. The application host cannot open a TCP connection to db01:5432. In order: is PostgreSQL running; is it bound to an address the app host can reach; does pg_hba.conf permit the app host’s network; is there a firewall. The health endpoint deliberately does not distinguish these, because a health endpoint that reports internal detail to unauthenticated callers is its own problem.

The service restarts in a loop. journalctl -u estate-app -n 50. With Restart=on-failure and RestartSec=2, a Python traceback produces a restart every two seconds and a unit that eventually hits systemd’s start limit. ProtectSystem=strict is a common cause: the process cannot write anywhere not listed in ReadWritePaths.

Backends show DOWN although curl from the controller works. The proxy checks from lb01, not from your controller. Check from lb01 itself, and confirm the proxy is checking the port the application actually binds.

socat reports “Connection refused” on the admin socket. HAProxy did not start, or started from a configuration without the stats socket line. systemctl status haproxy and head -5 /etc/haproxy/haproxy.cfg.

show stat output has fewer columns than the awk expects. Field positions in HAProxy’s CSV are stable across 2.x and 3.x, but a build with extra fields enabled shifts nothing — the columns are appended. If your awk finds nothing, print the header line first: echo "show stat" | socat stdio /run/haproxy/admin.sock | head -1.

Cleanup

Lab 3 and lab 4 both build on this deployment. Do not run Cleanup if you are continuing.

If you are stopping, tear down in reverse dependency order: proxy first so nothing is routed to a service you are about to stop, then the application, then the database.

Step 1. Stop routing:

$ cd "$HOME/estate"
ansible -i inventories/production/hosts.yml loadbalancers -b \
-m systemd_service -a 'name=haproxy state=stopped enabled=no'

Step 2. Restore the proxy configuration you captured, or remove ours:

$ ansible -i inventories/production/hosts.yml loadbalancers -b -m shell \
-a 'if [ -f /etc/haproxy/haproxy.cfg.pre-capstone ]; then
      cp -a /etc/haproxy/haproxy.cfg.pre-capstone /etc/haproxy/haproxy.cfg
      echo restored
    else
      rm -f /etc/haproxy/haproxy.cfg
      echo removed
    fi'

Step 3. Remove the application:

$ ansible -i inventories/production/hosts.yml appservers -b \
-m systemd_service -a 'name=estate-app state=stopped enabled=no'

ansible -i inventories/production/hosts.yml appservers -b -m file \
-a 'path=/etc/systemd/system/estate-app.service state=absent'

ansible -i inventories/production/hosts.yml appservers -b \
-m systemd_service -a 'daemon_reload=yes'

ansible -i inventories/production/hosts.yml appservers -b -m file \
-a 'path=/opt/estate-app state=absent'

ansible -i inventories/production/hosts.yml appservers -b -m file \
-a 'path=/var/log/estate-app state=absent'

Step 4. Remove the database objects. Read the warning first.

$ ansible -i inventories/production/hosts.yml databases -b \
--become-user postgres -m command -a 'dropdb --if-exists estate'

ansible -i inventories/production/hosts.yml databases -b \
--become-user postgres -m command \
-a 'psql -c "DROP ROLE IF EXISTS estate_app"'

Step 5. Remove the configuration drop-in and restore the cluster to its captured state:

$ ansible -i inventories/production/hosts.yml databases -b -m shell \
-a 'rm -f /etc/postgresql/*/main/conf.d/estate.conf'

ansible -i inventories/production/hosts.yml databases -b \
-m systemd_service -a 'name=postgresql state=restarted'

ansible -i inventories/production/hosts.yml databases -b \
-m command -a 'pg_isready'

Step 6. Confirm the estate is back to the shape the capture recorded:

$ ansible -i inventories/production/hosts.yml estate -b \
-m command -a 'ss -lntp'

grep -A20 'listening' reports/pre-deploy/lb01.yml

Leave the packages installed. Removing postgresql or haproxy is a package-manager operation that can pull out shared dependencies, and the lab’s own contract is to restore configuration and services, not to reverse an installation. If you need the machines truly pristine, the VM snapshot from lab 1 is the mechanism.

What You Learned

• A role that branches on the environment has moved a reviewable decision into an unreviewable place. Nine values in group_vars is what a comparable staging environment costs.
• validate: runs on the managed node against a temporary file, and only moves it into place on success — which is why a failed validation leaves the previous configuration completely intact.
• Not every daemon has an offline validator. When there is none, the validation is a post-activation assertion you write yourself, and pretending otherwise is worse than admitting it.
• Restart versus reload is a property of the setting, not a preference. listen_addresses needs a restart; a reload reports success and changes nothing, and the symptom points at the network.
• Handlers run in definition order, not notification order — which is what makes daemon-reload reliably precede a restart.
• A health check that has never failed has never been tested. Stopping the database and keeping the failing recap is the evidence.
• The health-check document’s most valuable section is what it does not cover, because a gate believed to cover more than it does converts “we did not check” into “we checked and it was fine”.