Lab: Design HA

Objective

By the end of this lab you will have a running stack in which one half is genuinely highly available and the other half is not, and you will be able to prove which is which by counting things rather than by reading a dashboard. Concretely: the Alertmanager pair will survive the loss of one node with exactly one notification per alert group, and the Prometheus pair will not survive the loss of one replica in any sense that matters, because the data on the dead replica was never anywhere else.

The point is not that Prometheus replication is hard. The point is that the two halves of this stack look identical from a process-count dashboard — two processes, both healthy, both green — and are opposite in behaviour. You will finish with a one-page statement of which is which and the measurement behind each line.

Architecture

One host, six containers, one Docker network. Both Prometheus replicas scrape the same target set and send alerts to both Alertmanagers; both Alertmanagers gossip with each other and deliver to the same webhook sink.

                    +---------------------+
                    |  node-exporter      |  <-- the shared target
                    |  node:9100          |      (scraped twice)
                    +----+-----------+----+
                         ^           ^
             scrape 15s  |           |  scrape 15s
                         |           |
        +----------------+--+     +--+-----------------+
        | prom-a             |     | prom-b            |
        | external_labels:   |     | external_labels:  |
        |   cluster=lab      |     |   cluster=lab     |
        |   replica=a        |     |   replica=b       |
        | TSDB A (own WAL)   |     | TSDB B (own WAL)  |
        +---+------------+---+     +---+-----------+---+
            |            |             |           |
            |            +-------------|-----------+
            |                          |           |
            v                          v           v
      +-----+--------+   gossip   +----+---------+
      | am-a :9093   | <========> | am-b :9093   |
      |        :9094 |   :9094    |        :9094 |
      +------+-------+            +------+-------+
             |                           |
             +------------+--------------+
                          v
                  +-------+--------+
                  | sink (nginx)   |  <-- counts notifications
                  | :8080/notify   |
                  +----------------+

Two things in that diagram carry the whole lab. The Prometheus replicas share nothing: two TSDBs, two WALs, two independent series stores, and no protocol between them. The Alertmanagers share everything that matters: a gossip channel on port 9094 over which they replicate alert state and the notification log. Same picture, opposite properties.

Requirements

• A Linux host (VM or workstation) with Docker Engine 28.x and Docker Compose v2. Declare B-nested if you run it in a VM; the lab does not need nested virtualisation itself, only a working container runtime.
• curl and jq on the host. Every measurement in this lab is an HTTP query parsed with jq. Without jq you will be reading raw JSON.
• Roughly 1 GiB of free memory and 2 GiB of disk. The images total around 500 MiB.
• Free TCP ports on the host: 9090, 9091, 9093, 9100, 9193, 18080. If any are taken, change the left-hand side of the port mappings in the compose file; nothing inside the lab depends on the host-side numbers.
• No out-of-band access requirement. This lab does not touch the host’s networking, firewall or SSH daemon. Everything it creates lives in one directory and one Docker Compose project, and Cleanup removes both.

Scenario

You have inherited a monitoring stack described in the runbook as “HA”. There are two Prometheus servers and two Alertmanagers. The wiki page says “redundancy: 2x on both tiers”. Last quarter a hypervisor host was rebooted for a kernel upgrade; it carried one Prometheus and one Alertmanager. Pages kept arriving during the reboot, so the change was recorded as a successful HA test.

Three weeks later somebody asked for a chart of API latency over the previous six months and the chart had a nine-hour hole in it — the hypervisor reboot plus the rebuild. Nobody could explain why the alerting tier survived and the metrics tier did not, because both tiers had “two of everything”.

Your job is to build the same shape on one host, small enough to reason about, and to come out with the specific answer: which tier is HA, for what failure, and what it cost.

Tasks

Task 1: Create the working directory and the configuration

Everything the lab creates lives under one directory so Cleanup is exact.

# Substitute your own path if $HOME is not writable:
LABDIR="$HOME/rb-obs-ha"

mkdir -p "$LABDIR"
cd "$LABDIR"

The two Prometheus configurations differ in exactly one line — the replica external label. Write prometheus-a.yml:

# prometheus-a.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    cluster: lab
    replica: a

rule_files:
  - /etc/prometheus/rules.yml

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - 'am-a:9093'
            - 'am-b:9093'

scrape_configs:
  - job_name: node
    static_configs:
      - targets: ['node:9100']

  - job_name: prometheus
    static_configs:
      - targets: ['prom-a:9090', 'prom-b:9090']

  - job_name: alertmanager
    static_configs:
      - targets: ['am-a:9093', 'am-b:9093']

Note that each Prometheus is configured to send alerts to both Alertmanagers. That is the documented pattern and it is deliberate: it is what makes the Alertmanager cluster’s deduplication load-bearing rather than decorative. Four alert deliveries per rule evaluation (two Prometheus servers times two Alertmanagers) have to collapse into one page.

Create prometheus-b.yml as a copy with the replica label changed:

sed 's/^    replica: a$/    replica: b/' prometheus-a.yml > prometheus-b.yml
grep -n 'replica:' prometheus-a.yml prometheus-b.yml

The grep is the check: two files, one line each, values a and b. If both say a, the sed did not match and every measurement later in this lab will be wrong in a way that looks like a working system.

Now the rules. rules.yml:

# rules.yml
groups:
  - name: lab-ha
    interval: 15s
    rules:
      # Deterministic: fires immediately and never stops. This is the alert we
      # count notifications for. vector(1) is always true, so the alert's
      # lifecycle is not entangled with whatever the host is doing.
      - alert: LabAlwaysFiring
        expr: vector(1)
        labels:
          severity: warning
        annotations:
          summary: 'Deterministic lab alert emitted by replica {{ $externalLabels.replica }}'

      # Real: fires when the shared target stops being scrapable.
      - alert: NodeExporterDown
        expr: up{job="node"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: 'node-exporter has been unscrapable for 1m'

{{ $externalLabels.replica }} resolves at notification time to the emitting Prometheus’s external label, so the annotation text tells you which replica a given copy of the alert came from without having to read the label set.

Task 2: Write the Alertmanager and sink configuration

alertmanager.yml — one file, used by both nodes:

# alertmanager.yml
route:
  receiver: sink
  # '...' is the documented special value meaning "group by every label".
  # It is a common production choice for teams that want one page per distinct
  # alert. It is also the setting that turns a replica label into a second page.
  group_by: ['...']
  group_wait: 10s
  group_interval: 30s
  repeat_interval: 1h

receivers:
  - name: sink
    webhook_configs:
      - url: 'http://sink:8080/notify'
        send_resolved: true

The sink is an nginx that accepts any POST to /notify, returns 204, and logs one line per request. It is the cheapest honest way to count notifications: the count is produced by the receiver, not by the thing being tested. sink.conf:

log_format notify '$time_iso8601 $request_method $uri bytes=$request_length';

server {
    listen 8080;
    access_log /dev/stdout notify;

    location /notify {
        return 204;
    }

    location / {
        return 404;
    }
}

Task 3: Write the compose file and start the stack

compose.yaml:

name: rb-obs-ha

services:
  node:
    image: prom/node-exporter:v1.8.2
    container_name: rb-ha-node
    ports:
      - '9100:9100'

  prom-a:
    image: prom/prometheus:v2.55.1
    container_name: rb-ha-prom-a
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=2h'
      - '--web.enable-lifecycle'
    volumes:
      - ./prometheus-a.yml:/etc/prometheus/prometheus.yml:ro
      - ./rules.yml:/etc/prometheus/rules.yml:ro
      - prom-a-data:/prometheus
    ports:
      - '9090:9090'

  prom-b:
    image: prom/prometheus:v2.55.1
    container_name: rb-ha-prom-b
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=2h'
      - '--web.enable-lifecycle'
    volumes:
      - ./prometheus-b.yml:/etc/prometheus/prometheus.yml:ro
      - ./rules.yml:/etc/prometheus/rules.yml:ro
      - prom-b-data:/prometheus
    ports:
      - '9091:9090'

  am-a:
    image: prom/alertmanager:v0.27.0
    container_name: rb-ha-am-a
    command:
      - '--config.file=/etc/alertmanager/alertmanager.yml'
      - '--storage.path=/alertmanager'
      - '--web.listen-address=0.0.0.0:9093'
      - '--cluster.listen-address=0.0.0.0:9094'
      - '--cluster.peer=am-a:9094'
      - '--cluster.peer=am-b:9094'
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
      - am-a-data:/alertmanager
    ports:
      - '9093:9093'

  am-b:
    image: prom/alertmanager:v0.27.0
    container_name: rb-ha-am-b
    command:
      - '--config.file=/etc/alertmanager/alertmanager.yml'
      - '--storage.path=/alertmanager'
      - '--web.listen-address=0.0.0.0:9093'
      - '--cluster.listen-address=0.0.0.0:9094'
      - '--cluster.peer=am-a:9094'
      - '--cluster.peer=am-b:9094'
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
      - am-b-data:/alertmanager
    ports:
      - '9193:9093'

  sink:
    image: nginx:1.27-alpine
    container_name: rb-ha-sink
    volumes:
      - ./sink.conf:/etc/nginx/conf.d/default.conf:ro
    ports:
      - '18080:8080'

volumes:
  prom-a-data:
  prom-b-data:
  am-a-data:
  am-b-data:

The --cluster.peer list is identical on both Alertmanagers, which is the documented pattern: the list is a set of seeds to contact, not a description of the cluster. A node contacts the seeds, learns the full membership, and gossips with everyone from then on.

$ docker compose up -d

Wait for the stack to settle, then confirm every container is running:

docker compose ps --format 'table {{.Name}}\t{{.Service}}\t{{.Status}}'

If prom-a or prom-b is restarting, its config failed to parse. Read the reason before continuing:

docker compose logs --tail=20 prom-a

Task 4: Confirm the pair is a duplicate-scrape pair, not two systems

Two questions: are both replicas scraping the same targets, and is anything distinguishing their output.

for port in 9090 9091; do
  echo "--- prometheus on :$port"
  curl -sG "http://localhost:$port/api/v1/query" \
    --data-urlencode 'query=up{job="node"}' \
  | jq -r '.data.result[] | "\(.metric.instance) replica=\(.metric.replica) up=\(.value[1])"'
done

Both should report the same instance with up=1, differing only in replica. That is the definition of a duplicate-scrape pair: identical internal labels, one distinguishing external label. If the replica field is empty, the external labels are not being applied and you should revisit Task 1.

Now the cost side of that arrangement, measured at the target rather than at either Prometheus. node_exporter instruments its own metrics handler, so the target itself will tell you how often it is being scraped:

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=rate(promhttp_metric_handler_requests_total{job="node",code="200"}[5m])' \
| jq -r '.data.result[] | "\(.value[1]) requests/sec"'

Let this run for five minutes after startup so the rate window is full. With a 15s scrape interval and one scraper the rate is one request every 15 seconds, about 0.067 per second. With two scrapers it is about 0.133 per second. That number is the price of the second replica, charged to every target in the estate, and it is the number that is missing from most “let’s add HA” proposals.

Task 5: Prove the Alertmanager cluster has formed

The gossip cluster is the half of this stack that is genuinely HA, so establish that it is actually clustered before testing it.

for port in 9093 9193; do
  echo "--- alertmanager on :$port"
  curl -s "http://localhost:$port/api/v2/status" \
  | jq '{status: .cluster.status, name: .cluster.name,
         peers: [.cluster.peers[].address]}'
done

Each node must report "status": "ready" and two peer addresses — itself and the other node. A node reporting "ready" with one peer is a single-node cluster that happens to be running next to another single-node cluster, which is precisely the anti-pattern this lab exists to make visible.

Cross-check against the metrics both nodes expose about their own membership:

for port in 9093 9193; do
  echo "--- alertmanager on :$port"
  curl -s "http://localhost:$port/metrics" | grep '^alertmanager_cluster'
done

Read the whole family rather than one metric name. The membership gauge, the health score and the peer join/leave counters are all in there, and reading them together tells you whether the cluster is merely formed or actually settled.

Task 6: Count the notifications — the double-page demonstration

LabAlwaysFiring has been firing since the stack started. Two Prometheus servers evaluate it; each sends its copy to both Alertmanagers. Ask the cluster what it thinks is going on:

curl -s http://localhost:9093/api/v2/alerts \
| jq -r '.[] | select(.labels.alertname == "LabAlwaysFiring")
         | "\(.labels.alertname) replica=\(.labels.replica) cluster=\(.labels.cluster)"'

Two lines: replica=a and replica=b. Alertmanager deduplicates on the exact label set, and these two label sets are not equal, so it is holding two alerts. It is not confused; it is doing what it was told.

Now count what actually left the building:

docker compose logs sink | grep -c 'POST /notify'

With group_by: ['...'] — group by every label — the two alerts land in two different aggregation groups, so the sink has received two notifications for one condition. That is the double page from lesson 03, reproduced on your own host with a number attached.

Confirm the cluster is nonetheless doing its job by checking that only one node sent each notification:

for port in 9093 9193; do
  echo "--- alertmanager on :$port"
  curl -s "http://localhost:$port/metrics" \
  | grep '^alertmanager_notifications_total{.*integration="webhook"'
done

The two counters sum to the sink’s count; they do not each equal it. Each notification was sent once by one node, and the notification log gossiped to the other node stopped it sending a second copy. The cluster deduplicated across nodes. It did not deduplicate across replicas, because nobody asked it to.

Task 7: Fix the double page at its cause

There are two ways to make the sink receive one notification. Only one of them is a fix.

The tempting one is to change group_by so replica is not a grouping key — say group_by: ['alertname', 'severity']. The sink then receives one notification, because both alerts land in one group. But Alertmanager is still holding two alerts, the UI still shows two, any receiver that renders one message per alert in the group still produces two lines, and the silence you write against one of them does not match the other. The count improved; the duplication did not.

The fix is to stop the two copies being distinguishable by the time they reach Alertmanager. Prometheus applies alert_relabel_configs to alerts on their way out, after rule evaluation and before delivery. Drop the replica label there. Add this block to both Prometheus configs, inside alerting::

alerting:
  alert_relabel_configs:
    # Strip the replica identity from outgoing alerts. The series in the local
    # TSDB keep it; only the alert loses it. Both replicas now emit a
    # byte-identical label set, so Alertmanager's own dedup collapses them.
    - regex: replica
      action: labeldrop
  alertmanagers:
    - static_configs:
        - targets:
            - 'am-a:9093'
            - 'am-b:9093'

Apply it to both files, reload both servers, and clear the sink’s history so the count that follows is unambiguous:

# Both Prometheus servers run with --web.enable-lifecycle, so a POST to
# /-/reload re-reads the config without restarting the process or the TSDB.
for port in 9090 9091; do
  curl -sf -X POST "http://localhost:$port/-/reload" && echo "reloaded :$port"
done

Confirm the reload took effect before you count anything:

for port in 9090 9091; do
  curl -s "http://localhost:$port/api/v1/status/config" \
  | jq -r '.data.yaml' | grep -A3 'alert_relabel_configs' || echo ":$port NOT reloaded"
done

Existing alerts keep the label set they were created with, so restart the Alertmanagers to start from an empty state, then wait out group_wait plus one group_interval — about 45 seconds — and count again:

$ docker compose restart am-a am-b

curl -s http://localhost:9093/api/v2/alerts \
| jq -r '[.[] | select(.labels.alertname == "LabAlwaysFiring")] | length'

One alert, not two. The replica label is gone from the alert and present in the metrics — check that second half, because losing it from the TSDB as well would be a real regression:

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=count by (replica) (up)' \
| jq -r '.data.result[] | "replica=\(.metric.replica) series=\(.value[1])"'

Task 8: Kill one node of each tier and watch the difference

This is the test that lesson 05 calls evidence. Do the alerting tier first, because it is the one that passes.

$ docker compose stop am-a

# Wait for the survivor to notice, then read its view of the cluster.
sleep 20
curl -s http://localhost:9193/api/v2/status \
| jq '{status: .cluster.status, peers: [.cluster.peers[].address]}'

The survivor reports itself and, for a short while, the departed node until the failure detector converges. Notifications continue: the surviving node’s position in the member list moved to zero, so it dispatches without waiting. Check that the sink is still receiving by watching for new lines:

docker compose logs --since=2m sink | grep 'POST /notify' | tail -5

Bring it back and confirm the cluster re-forms:

docker compose start am-a
sleep 20
curl -s http://localhost:9093/api/v2/status | jq '.cluster.status'

Now the metrics tier. Record a baseline, stop one replica for three minutes, start it again, and look for the repair that never comes:

docker compose stop prom-b
sleep 180
docker compose start prom-b
sleep 60

# The same question asked of both replicas: how many scrapes of the shared
# target do you hold in the last fifteen minutes?
for port in 9090 9091; do
  printf ':%s ' "$port"
  curl -sG "http://localhost:$port/api/v1/query" \
    --data-urlencode 'query=count_over_time(up{job="node"}[15m])' \
  | jq -r '.data.result[0].value[1]'
done

prom-a reports roughly 60 samples — fifteen minutes at a 15-second interval. prom-b reports roughly 48, because it was not running for twelve of those intervals. Wait ten minutes and ask again. The numbers do not converge. There is no backfill, no catch-up, no anti-entropy pass. prom-b has a hole in its history and will have that hole until the samples age out of retention.

Validation

Four checks. Each proves an outcome rather than restating a step.

1. The Alertmanager tier is HA for single-node loss. With one node stopped, notifications continue to arrive at the sink:

docker compose stop am-a
sleep 90
docker compose logs --since=60s sink | grep -c 'POST /notify'
docker compose start am-a

A non-zero count with a node down is the evidence. Zero means the surviving node is not taking over and the cluster was never real.

2. Cross-replica duplication is gone at the cause, not hidden. Alertmanager holds one alert, and it holds it under a label set with no replica key:

curl -s http://localhost:9093/api/v2/alerts \
| jq -r '[.[] | select(.labels.alertname == "LabAlwaysFiring")]
         | "alerts=\(length) has_replica_label=\(.[0].labels | has("replica"))"'

Expected: alerts=1 has_replica_label=false.

3. The metrics tier is not HA, and the proof is a number that does not converge. Run the sample-count comparison from Task 8 twice, ten minutes apart. Both runs show prom-b behind prom-a by the same absolute amount. A shrinking gap would mean something was replicating; nothing is.

4. The cost of the pair is measured, not assumed.

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=rate(promhttp_metric_handler_requests_total{job="node",code="200"}[10m]) * 60' \
| jq -r '.data.result[] | "\(.value[1]) scrapes/min against the shared target"'

Roughly 8 per minute for two replicas at 15s. Halve that number and you have the single-replica baseline; the difference is what the availability cost.

Expected Outcome

• Six containers running; both Prometheus replicas scraping the same target and distinguishable only by their replica external label.
• Both Alertmanagers reporting "ready" with two peers, and surviving the loss of either node with notification delivery unbroken.
• Exactly one alert in Alertmanager for LabAlwaysFiring, carrying no replica label, produced by alert_relabel_configs rather than by grouping.
• A measured scrape multiplier against the shared target of approximately 2x.
• A permanent, measured gap in prom-b’s history that no mechanism repairs.

Troubleshooting

• prom-a restarts in a loop. The config failed to parse. docker compose logs prom-a names the line. The usual cause is indentation under external_labels or a tab character introduced by an editor.
• Both replicas report replica=a. The sed in Task 1 did not match, probably because the file was edited by hand with different indentation. Edit prometheus-b.yml directly and reload.
• /api/v2/status returns 404. You are on an Alertmanager older than the v2 API, or you typed /api/v1/. Version 0.27 has removed v1.
• Cluster status is settling and stays there. The nodes cannot reach each other on 9094. Confirm both are on the same compose network with docker compose exec am-a wget -qO- http://am-b:9093/-/ready.
• No lines in the sink log. Alertmanager cannot resolve sink. Check the receiver URL uses the compose service name, not localhost — inside the network, localhost is the Alertmanager container itself.
• The reload returns 404. --web.enable-lifecycle is missing from that container’s command. Add it and recreate the container; a reload cannot enable the endpoint that performs reloads.
• Sample counts look identical after the outage. The 15m window has already slid past the outage. Re-run the comparison sooner after restarting prom-b, or widen the window to [30m].

Cleanup

Cleanup removes the containers, the four named volumes and the working directory. Nothing was installed on the host and no host configuration was changed, so this returns the machine exactly to its prior state.

$ cd ~/rb-obs-ha && docker compose down -v

# Confirm nothing from the project survives, then remove the directory.
docker compose ls --all | grep rb-obs-ha || echo "project gone"
docker volume ls | grep rb-obs-ha || echo "volumes gone"

rm -rf "$HOME/rb-obs-ha"

The images remain in the local cache. Remove them only if you want the disk back:

docker image rm prom/prometheus:v2.55.1 prom/alertmanager:v0.27.0 \
  prom/node-exporter:v1.8.2 nginx:1.27-alpine

Production notes

Mapping this exercise onto a real change window:

The Alertmanager change is a low-risk rolling change. Adding --cluster.peer to an existing pair and restarting them one at a time is a few seconds of unavailability per node, absorbed by the peer. Do it during business hours with somebody watching, not at 02:00 alone. The check that it worked is /api/v2/status showing the expected peer count on every node — verify it on each node individually, because a node that cannot reach the others reports itself healthy.

The alert_relabel_configs change is a config reload, and it is not reversible in the alert store. Alerts created before the reload keep their old label set until they resolve, so for one alert lifetime you will see both shapes. Any silence written against the old label set stops matching once the new shape arrives. Audit active silences for replica= matchers before you make this change, and rewrite them first.

The Prometheus finding is a design conversation, not a change. Nothing in this lab makes a Prometheus pair highly available for its data, because nothing can: two Prometheus processes have no protocol for exchanging samples. The options are to ship both replicas’ samples into a store that does replicate and deduplicate on read, to shard the target set so no target is scraped twice and accept losing a shard when a replica dies, or to decide the exposure is acceptable and write down that the metrics tier has an RPO equal to its retention window. All three are defensible. Silently believing the pair is HA is not.

What You Learned

• A duplicate-scrape pair is one distinguishing label away from working and one label away from double paging. The replica external label is what makes the two copies addressable, and it is what has to be stripped from alerts before they reach Alertmanager.
• Alertmanager’s cluster deduplicates across nodes, never across label sets. The gossiped notification log stops the second node sending a second copy. It cannot know that two different label sets are the same incident.
• group_by changes the notification count without changing the duplication. Fixing the number you can see while leaving the cause in place is the shape of most “we fixed the alert noise” work.
• Two Prometheus processes have no replication protocol. The gap you measured in Task 8 is permanent, and no amount of load balancing, health checking or process supervision closes it.
• HA has a price and it is charged to the targets. You measured it: the shared exporter serves twice the requests. Any HA proposal that does not name that multiplier has not been costed.