Lab: Rolling deploy behind a real proxy, with drain and health gate

Objective

By the end of this lab you will have deployed a version bump across four application hosts with zero failed requests, proved that claim from the proxy’s own log rather than by assertion, and then deliberately broken the deployment on the second host and left the fleet in a state you can describe in one sentence to somebody who was not there.

Architecture

Five VMs: one proxy, four application hosts. The proxy is the thing that makes this lab real — a drain step against a fake load balancer teaches nothing.

                    client (curl loop)
                          │
                    ┌─────▼─────┐
                    │  haproxy  │  lb1  192.0.2.20
                    │  :80      │  admin socket /run/haproxy/admin.sock
                    └─────┬─────┘
          ┌───────────┬───┴───────┬───────────┐
       app1        app2         app3        app4
     192.0.2.11  192.0.2.12   192.0.2.13  192.0.2.14
       :8080        :8080        :8080       :8080

Requirements

• A controller with ansible-core 2.21.x and the community.general collection, for the haproxy module.
• Five VMs with systemd as PID 1. The drain depends on a real HAProxy admin socket, the health gate on a real service that takes time to start, and the evidence on a real access log. B-nested only; there is no container path that keeps any of those honest.
• SSH key access and become on all five.
• Port 80 free on the proxy and 8080 free on the app hosts.
• No out-of-band access requirement: the lab does not reconfigure SSH, the firewall or networking. It does stop and start application services and will make a host serve 503s for a few seconds; Cleanup restores.

Scenario

You deploy a new version of an internal application to four hosts behind a load balancer. The current procedure is ansible-playbook deploy.yml, which restarts all four services within about two seconds of each other. Nobody has measured how many requests that drops, because nobody has looked at the proxy log.

Today you are going to look, and then to build the version that does not drop any.

Tasks

Task 1: Capture, and set up the estate

WORKDIR="$HOME/ansible-rolling-lab"
mkdir -p "$WORKDIR"/{templates,reports}
cd "$WORKDIR"

inventory.yml:

all:
  children:
    proxy:
      hosts:
        lb1: {ansible_host: 192.0.2.20}
    appservers:
      hosts:
        app1: {ansible_host: 192.0.2.11}
        app2: {ansible_host: 192.0.2.12}
        app3: {ansible_host: 192.0.2.13}
        app4: {ansible_host: 192.0.2.14}
  vars:
    ansible_user: operator
    app_port: 8080
    haproxy_socket: /run/haproxy/admin.sock
    haproxy_backend: app_backend

# capture.yml
- name: Back up the proxy configuration before we replace it
  hosts: proxy
  become: true
  gather_facts: false
  tasks:
    - name: Check for an existing configuration
      ansible.builtin.stat:
        path: /etc/haproxy/haproxy.cfg
      register: cfg

    - name: Back it up, once, without overwriting an earlier backup
      ansible.builtin.copy:
        src: /etc/haproxy/haproxy.cfg
        dest: /etc/haproxy/haproxy.cfg.pre-lab
        remote_src: true
        mode: preserve
        force: false
      when: cfg.stat.exists

    - name: Record whether there was one at all
      ansible.builtin.copy:
        content: "haproxy_cfg_existed: {{ cfg.stat.exists }}\n"
        dest: "{{ playbook_dir }}/reports/pre-lab-proxy.yml"
        mode: '0644'
      delegate_to: localhost
      become: false

Then a minimal application on each app host — a systemd unit serving its version string, which is all the lab needs to observe:

# setup-app.yml
- name: Install the lab application
  hosts: appservers
  become: true
  gather_facts: false
  vars:
    app_version: '1.0.0'
  tasks:
    - name: Install the application script
      ansible.builtin.copy:
        content: |
          #!/usr/bin/env python3
          import http.server, os, socketserver, sys
          VERSION = os.environ.get('APP_VERSION', 'unknown')
          HOST = os.uname().nodename
          class H(http.server.BaseHTTPRequestHandler):
              def do_GET(self):
                  if self.path == '/health':
                      self.send_response(200); self.end_headers()
                      self.wfile.write(b'ok\n')
                  else:
                      self.send_response(200); self.end_headers()
                      self.wfile.write(f'{HOST} {VERSION}\n'.encode())
              def log_message(self, *a): pass
          socketserver.TCPServer.allow_reuse_address = True
          with socketserver.TCPServer(('', int(sys.argv[1])), H) as s:
              s.serve_forever()
        dest: /usr/local/bin/labapp
        mode: '0755'

    - name: Install the environment file
      ansible.builtin.copy:
        content: "APP_VERSION={{ app_version }}\n"
        dest: /etc/default/labapp
        mode: '0644'

    - name: Install the unit
      ansible.builtin.copy:
        content: |
          [Unit]
          Description=Lab application
          After=network-online.target

          [Service]
          EnvironmentFile=/etc/default/labapp
          ExecStart=/usr/local/bin/labapp {{ app_port }}
          Restart=on-failure

          [Install]
          WantedBy=multi-user.target
        dest: /etc/systemd/system/labapp.service
        mode: '0644'

    - name: Start it
      ansible.builtin.systemd_service:
        name: labapp
        state: started
        enabled: true
        daemon_reload: true

Task 2: Configure the proxy with a real admin socket

templates/haproxy.cfg.j2:

global
    log /dev/log local0
    stats socket {{ haproxy_socket }} mode 660 level admin
    stats timeout 30s

defaults
    log     global
    mode    http
    option  httplog
    timeout connect 5s
    timeout client  30s
    timeout server  30s

frontend fe_app
    bind *:80
    default_backend {{ haproxy_backend }}

backend {{ haproxy_backend }}
    balance roundrobin
    option httpchk GET /health
    http-check expect status 200
{% for h in groups['appservers'] %}
    server {{ h }} {{ hostvars[h].ansible_host }}:{{ app_port }} check inter 2s fall 2 rise 2
{% endfor %}

# setup-proxy.yml
- name: Configure the proxy
  hosts: proxy
  become: true
  gather_facts: false
  tasks:
    - name: Render the configuration, validating before activation
      ansible.builtin.template:
        src: haproxy.cfg.j2
        dest: /etc/haproxy/haproxy.cfg
        mode: '0644'
        backup: true
        validate: 'haproxy -c -f %s'
      notify: Reload haproxy

    - name: Ensure haproxy is running
      ansible.builtin.systemd_service:
        name: haproxy
        state: started
        enabled: true

  handlers:
    - name: Reload haproxy
      ansible.builtin.systemd_service:
        name: haproxy
        state: reloaded

$ ansible-playbook -i inventory.yml capture.yml setup-app.yml setup-proxy.yml

Confirm the estate works:

# Substitute your own values before running:
VIP=192.0.2.20

for i in $(seq 1 8); do curl -s "http://$VIP/"; done

You should see all four hostnames, round-robined, all reporting 1.0.0.

Task 3: Measure the naive deployment

Start a client loop on the controller, in another terminal:

# Substitute your own values before running:
VIP=192.0.2.20
OUT="$HOME/ansible-rolling-lab/reports/client-naive.txt"

while true; do
  code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 2 "http://$VIP/")
  printf '%s %s\n' "$(date +%s.%N)" "$code" >> "$OUT"
  sleep 0.05
done

Then run the deployment as it exists today — all four at once:

# deploy-naive.yml
- name: Deploy without draining
  hosts: appservers
  become: true
  gather_facts: false
  vars:
    app_version: '2.0.0'
  tasks:
    - name: Write the new version
      ansible.builtin.copy:
        content: "APP_VERSION={{ app_version }}\n"
        dest: /etc/default/labapp
        mode: '0644'

    - name: Restart the application
      ansible.builtin.systemd_service:
        name: labapp
        state: restarted

$ ansible-playbook -i inventory.yml deploy-naive.yml

Stop the client loop and count:

cd "$HOME/ansible-rolling-lab"
awk '{print $2}' reports/client-naive.txt | sort | uniq -c

$ awk '{print $2}' reports/client-naive.txt | sort | uniq -c

    412 200
   37 503
    4 000

Thirty-seven 503s and four connection failures. HAProxy’s health check interval is 2 seconds with fall 2, so it takes up to four seconds to notice a backend is down — and all four went down inside that window.

Record the number. It is what the rest of the lab is measured against.

Task 4: Build the rolling deployment

# deploy-rolling.yml
- name: Rolling deployment with drain and health gate
  hosts: appservers
  become: true
  gather_facts: false
  serial: 1
  max_fail_percentage: 0

  vars:
    app_version: '2.0.0'
    drain_settle_seconds: 5
    health_retries: 20
    health_delay: 2

  tasks:
    - name: Announce the host
      ansible.builtin.debug:
        msg: "deploying {{ app_version }} to {{ inventory_hostname }}"

    - name: Drain this host from the proxy
      community.general.haproxy:
        state: drain
        host: "{{ inventory_hostname }}"
        backend: "{{ haproxy_backend }}"
        socket: "{{ haproxy_socket }}"
        wait: true
        wait_interval: 1
        wait_retries: 30
      delegate_to: "{{ groups['proxy'][0] }}"
      become: true

    - name: Let in-flight requests finish
      ansible.builtin.wait_for:
        timeout: "{{ drain_settle_seconds }}"
      delegate_to: localhost
      become: false

    - name: Write the new version
      ansible.builtin.copy:
        content: "APP_VERSION={{ app_version }}\n"
        dest: /etc/default/labapp
        mode: '0644'

    - name: Restart the application
      ansible.builtin.systemd_service:
        name: labapp
        state: restarted

    - name: Wait for the port to accept connections
      ansible.builtin.wait_for:
        host: "{{ ansible_host }}"
        port: "{{ app_port }}"
        state: started
        timeout: 30
      delegate_to: localhost
      become: false

    - name: Health-gate on the host itself, not through the proxy
      ansible.builtin.uri:
        url: "http://{{ ansible_host }}:{{ app_port }}/health"
        status_code: 200
        return_content: true
      register: health
      retries: "{{ health_retries }}"
      delay: "{{ health_delay }}"
      until: health.status == 200
      delegate_to: localhost
      become: false

    - name: Confirm the host is serving the version we deployed
      ansible.builtin.uri:
        url: "http://{{ ansible_host }}:{{ app_port }}/"
        return_content: true
      register: served
      delegate_to: localhost
      become: false
      failed_when: app_version not in served.content

    - name: Return the host to service
      community.general.haproxy:
        state: enabled
        host: "{{ inventory_hostname }}"
        backend: "{{ haproxy_backend }}"
        socket: "{{ haproxy_socket }}"
        wait: true
        wait_interval: 1
        wait_retries: 30
      delegate_to: "{{ groups['proxy'][0] }}"
      become: true

    - name: Record the outcome for this host
      ansible.builtin.copy:
        content: |
          host: {{ inventory_hostname }}
          deployed: {{ app_version }}
          served: {{ served.content | trim }}
          at: {{ lookup('pipe', 'date -Is') }}
        dest: "{{ playbook_dir }}/reports/deployed-{{ inventory_hostname }}.yml"
        mode: '0644'
      delegate_to: localhost
      become: false

Task 5: Prove the claim from the proxy log

Restart the client loop, run the rolling deployment, and then check both sides.

# Substitute your own values before running:
VIP=192.0.2.20
OUT="$HOME/ansible-rolling-lab/reports/client-rolling.txt"

Run the same loop as Task 3 with the new output file, deploy, then:

cd "$HOME/ansible-rolling-lab"
awk '{print $2}' reports/client-rolling.txt | sort | uniq -c

Expected: 200 and nothing else.

Now the stronger evidence — the proxy’s own view. HAProxy’s httplog records which backend server answered each request:

# Substitute your own values before running:
PROXY=lb1

ansible -i inventory.yml "$PROXY" -b -m shell -a \
  'journalctl -u haproxy --since "10 min ago" --no-pager | grep -oE "app_backend/[a-z0-9]+" | sort | uniq -c'

$ ansible -i inventory.yml lb1 -b -m shell -a 'journalctl -u haproxy --since \"10 min ago\" --no-pager | grep -oE \"app_backend/[a-z0-9]+\" | sort | uniq -c'

lb1 | CHANGED | rc=0 >>
  218 app_backend/app1
  229 app_backend/app2
  224 app_backend/app3
  221 app_backend/app4

Every request has a named backend and none has app_backend/<NOSRV>, which is what HAProxy logs when it had no server available. That, plus zero 503s at the client, is the proof.

Save both to reports/. This pair of artefacts is what a deployment review should require and almost never does.

Task 6: Break it midway

Add a deliberately broken deployment for one host:

    - name: Write the new version
      ansible.builtin.copy:
        content: |
          APP_VERSION={{ app_version }}
          {% if inventory_hostname == 'app2' %}
          BROKEN=1
          {% endif %}
        dest: /etc/default/labapp
        mode: '0644'

…and make the application refuse to start when BROKEN is set, by adding a ExecStartPre=/bin/sh -c '[ -z "$BROKEN" ]' to the unit.

Run the rolling deployment with app_version: 3.0.0 and a client loop going.

$ ansible-playbook -i inventory.yml deploy-rolling.yml -e app_version=3.0.0

PLAY [Rolling deployment with drain and health gate] ***************************
TASK [Announce the host] *******************************************************
ok: [app1] => {"msg": "deploying 3.0.0 to app1"}
...
TASK [Return the host to service] **********************************************
changed: [app1]

PLAY [Rolling deployment with drain and health gate] ***************************
TASK [Drain this host from the proxy] ******************************************
changed: [app2 -> lb1]

TASK [Health-gate on the host itself, not through the proxy] *******************
FAILED - RETRYING: Health-gate on the host itself (20 retries left)
...
fatal: [app2]: FAILED! => {"attempts": 20, "msg": "Connection refused"}

NO MORE HOSTS LEFT *************************************************************

PLAY RECAP *********************************************************************
app1                       : ok=11   changed=4    unreachable=0    failed=0
app2                       : ok=5    changed=3    unreachable=0    failed=1

The client loop should still show only 200s: app2 was drained before it broke, so no request ever reached the broken process. That is the whole value of the ordering.

Task 7: Land the fleet in a known state

Establish the truth before deciding anything:

# Substitute your own values before running:
PROXY=lb1

# What does the proxy think?
ansible -i inventory.yml "$PROXY" -b -m shell -a \
  "echo 'show servers state' | socat stdio /run/haproxy/admin.sock"

# What is each host actually serving?
ansible -i inventory.yml appservers -b -m shell -a \
  'curl -sf http://127.0.0.1:8080/ || echo "NOT SERVING"'

The state is: app1 on 3.0.0 and in service; app2 broken and drained; app3 and app4 on 2.0.0 and in service.

You have three options and each is defensible under different circumstances. Write your choice and its reasoning in reports/decision.md:

1. Roll forward. Fix the defect on app2, complete the deployment. Right when the defect is understood, host-specific and quick to fix. Leaves the fleet on mixed versions for as long as that takes.
2. Roll back app1. Return the whole fleet to 2.0.0, leave app2 drained, investigate offline. Right when the defect might not be host-specific — you do not yet know that app3 would have survived.
3. Leave it. Three hosts serving, one drained, mixed versions. Right only if 2.0.0 and 3.0.0 are genuinely compatible and there is a reason not to act now.

Execute your choice, then re-run the evidence commands and confirm the fleet matches what you wrote down.

Validation

• The naive deployment produces a non-zero count of 503 or 000 responses in reports/client-naive.txt.
• The rolling deployment produces only 200 responses in reports/client-rolling.txt.
• The HAProxy log for the rolling window contains no <NOSRV> entries.
• Each host’s reports/deployed-*.yml records the version it served, confirmed by an HTTP request to the host directly.
• With the induced failure, app2 fails the health gate after its retries, the play aborts, and the client loop still shows only 200s.
• show servers state on the proxy reports app2 in a drained state after the abort.
• reports/decision.md names one of the three options, its reasoning, and the fleet state you verified afterwards.

Expected Outcome

ansible-rolling-lab/
├── capture.yml, setup-app.yml, setup-proxy.yml
├── deploy-naive.yml, deploy-rolling.yml
├── inventory.yml
├── reports/
│   ├── client-naive.txt, client-rolling.txt
│   ├── decision.md
│   ├── deployed-app{1..4}.yml
│   └── pre-lab-proxy.yml
└── templates/haproxy.cfg.j2

Four hosts on a consistent version, all in service at the proxy, and a pair of measurements — client response codes and proxy backend attribution — that turn “we did a rolling deploy” into a number.

Troubleshooting

community.general.haproxy fails with a socket permission error. The socket needs level admin in the stats socket line, and the task needs become: true on the proxy. Check both; the error message distinguishes them poorly.

The drain task runs on the app host instead of the proxy. delegate_to: "{{ groups['proxy'][0] }}" is what moves it. Without it, the module looks for the admin socket on the host being deployed, which does not have one.

wait: true on the drain returns immediately. HAProxy reports the server as drained as soon as the state is set; wait polls until the state is reached, not until connections finish. That is why the explicit settle step exists.

The health gate passes on a host that is broken. It is pointed at the VIP. This is the failure the callout in Task 4 is about, and it is worth re-reading the URL character by character.

retries has no effect. retries/delay require until:. Without it, the task runs once. This silently makes a health gate a single attempt against a service that needs three seconds to start.

The proxy log shows <NOSRV> during the rolling run. Two hosts were out at once — either a previous run left one drained, or the health check interval let HAProxy mark a second host down. show servers state before starting, every time.

journalctl -u haproxy shows no request lines. option httplog is missing from the defaults section, or the log is going to a file via rsyslog instead of the journal. grep haproxy /var/log/haproxy.log is the alternative.

Cleanup

This lab replaced the proxy configuration, installed a systemd unit on four hosts, and may have left a host drained. All three must be reversed, and the drained host is the one people forget.

Step 1. Return every backend to service, before anything else:

cd "$HOME/ansible-rolling-lab"

ansible -i inventory.yml lb1 -b -m shell -a \
  "for s in app1 app2 app3 app4; do echo \"set server app_backend/\$s state ready\" | socat stdio /run/haproxy/admin.sock; done"

ansible -i inventory.yml lb1 -b -m shell -a \
  "echo 'show servers state' | socat stdio /run/haproxy/admin.sock"

Step 2. Remove the application from the app hosts:

# cleanup-app.yml
- name: Remove the lab application
  hosts: appservers
  become: true
  gather_facts: false
  tasks:
    - name: Stop and disable the service
      ansible.builtin.systemd_service:
        name: labapp
        state: stopped
        enabled: false
      failed_when: false

    - name: Remove the unit, the script and the environment file
      ansible.builtin.file:
        path: "{{ item }}"
        state: absent
      loop:
        - /etc/systemd/system/labapp.service
        - /usr/local/bin/labapp
        - /etc/default/labapp

    - name: Reload systemd so the removed unit is forgotten
      ansible.builtin.systemd_service:
        daemon_reload: true

Step 3. Restore the proxy configuration, branching on whether one existed:

# cleanup-proxy.yml
- name: Restore the proxy configuration
  hosts: proxy
  become: true
  gather_facts: false
  tasks:
    - name: Load the capture
      ansible.builtin.include_vars:
        file: "{{ playbook_dir }}/reports/pre-lab-proxy.yml"
      delegate_to: localhost
      become: false

    - name: Look for the backup
      ansible.builtin.stat:
        path: /etc/haproxy/haproxy.cfg.pre-lab
      register: backup

    - name: Restore the original configuration when there was one
      ansible.builtin.copy:
        src: /etc/haproxy/haproxy.cfg.pre-lab
        dest: /etc/haproxy/haproxy.cfg
        remote_src: true
        mode: preserve
      when: haproxy_cfg_existed | bool and backup.stat.exists

    - name: Validate the restored configuration BEFORE restarting
      ansible.builtin.command: haproxy -c -f /etc/haproxy/haproxy.cfg
      register: check
      changed_when: false
      failed_when: false

    - name: Refuse to restart onto an invalid configuration
      ansible.builtin.assert:
        that: check.rc == 0
        fail_msg: >-
          Restored haproxy.cfg does not validate on {{ inventory_hostname }}.
          haproxy has NOT been restarted and is still running the previous
          configuration. Investigate before proceeding.

    - name: Restart haproxy onto the restored configuration
      ansible.builtin.systemd_service:
        name: haproxy
        state: restarted

    - name: Remove the backup and the lab's own backups
      ansible.builtin.shell: |
        set -euo pipefail
        rm -f /etc/haproxy/haproxy.cfg.pre-lab
        rm -f /etc/haproxy/haproxy.cfg.*~
      args:
        executable: /bin/bash
      changed_when: true

Step 4. Verify and remove the working directory:

ansible -i inventory.yml lb1 -b -m command -a 'systemctl is-active haproxy'
ansible -i inventory.yml appservers -b -m shell \
  -a 'systemctl list-unit-files labapp.service 2>/dev/null | wc -l'

mkdir -p "$HOME/ansible-lab-deliverables/rolling"
cp -a deploy-rolling.yml reports/decision.md templates/haproxy.cfg.j2 \
      "$HOME/ansible-lab-deliverables/rolling/"

rm -rf "$HOME/ansible-rolling-lab"

What You Learned

• You measured the naive deployment before improving it. Thirty-seven 503s is a number; “restarting all four at once seems bad” is not.
• Drain, settle, deploy, health-gate, return. The ordering is the design, and draining before the host can break is what kept the client loop clean even when the deployment failed.
• Health-check the host directly, never the VIP. A check against the proxy passes because the other three hosts are up, and returns a broken host to service.
• A 200 is not a version check. The second assertion — that the served content contains the version deployed — catches a restart that silently kept the old process.
• The proxy log is the proof. Zero <NOSRV> entries and a named backend for every request is evidence; the recap is not.
• delegate_to moves execution, not variable scope. inventory_hostname still names the host being deployed, which is what makes the drain task readable.
• Mixed versions after an abort is a decision. You wrote down which of three options you chose and why, and version-skew compatibility is a question to answer before the deployment rather than during it.