Lab: Configure Alertmanager

Objective

By the end of this lab you will have an alertmanager.yml that routes six different alerts to five different destinations, and evidence for every decision it makes: a routing table produced before deployment, a webhook sink that logs exactly what arrived, and a pair of experiments that show grouping and inhibition changing the number of notifications an incident produces without changing the alerts themselves.

The rule that fires is not the alert. From the moment Prometheus posts it, Alertmanager decides what it means to a human being. This lab is that machine, on a workbench, with the lid off.

Architecture

Two containers and a shell. There is no Prometheus in this lab, on purpose: alerts are posted straight into Alertmanager, so nothing that happens can be blamed on a rule, a scrape or a for: dwell.

  your shell
      |
      |  amtool alert add  ->  POST /api/v2/alerts
      v
  +---------------------------------------------+
  |  Alertmanager :9093                         |
  |                                             |
  |   inhibit  ->  silence  ->  route  ->  group|
  |                                             |
  +---------------------------------------------+
      |
      |  webhook POST, one per group per window
      v
  +---------------------------------------------+
  |  sink :8080  (25 lines of Python)           |
  |  prints receiver, groupLabels, alert count  |
  +---------------------------------------------+

The sink is the only honest way to answer “did the notification arrive, and what was in it”. The Alertmanager UI shows you what Alertmanager holds; the sink shows you what a receiver was actually sent, which is a different question and the one that matters.

Requirements

• Linux or macOS with a shell, Docker Engine 28.x and Docker Compose v2.
• TCP port 9093 free on the host. The sink is reachable only on the Compose network and publishes no port.
• curl and jq. amtool runs from the Alertmanager image.
• 90 minutes. Several steps wait out a group_wait or a group_interval; those waits are the measurement.
• No out-of-band access requirement. Nothing here touches host networking, SSH or the firewall.

Scenario

A database cluster goes dark at 02:40. ClusterDown fires once. HostDown fires on each of eight hosts. DiskFillingSoon was already firing on two of them at severity=warning. Eleven alerts, one incident, one root cause.

The team’s Alertmanager sends eleven notifications to one channel. The on-call engineer reads all eleven, works out they are the same event, and starts the actual investigation four minutes later than they could have. Nobody wrote a bad alert rule. Every one of those eleven alerts is correct. The defect is in the file that decides what to do with them.

Tasks

Task 1: Lay out the directory and the helpers

$ ss -ltnp 2>/dev/null | grep -E ':9093\b' || echo 'port free'

On macOS use lsof -nP -iTCP:9093 -sTCP:LISTEN instead.

WORKDIR="$HOME/obs-alertmanager-lab"
mkdir -p "$WORKDIR"
cd "$WORKDIR"

Two helpers. The first runs amtool against a config file with no server involved, which is what you want before anything is deployed. The second runs amtool against the live instance. Both are used throughout; re-declare them if you open a new terminal.

amtool_file() {
  docker run --rm -v "$PWD:/work" -w /work \
    --entrypoint /bin/amtool prom/alertmanager:v0.28.1 "$@"
}

amtool_live() {
  docker compose exec -T alertmanager \
    amtool --alertmanager.url=http://localhost:9093 "$@"
}

Task 2: Write the route tree

The shape below is the one to reach for by default: severity first, team second. The first split guarantees that a critical alert always pages somebody even if its team label is missing or misspelled. Split by team first and a critical alert with an unknown team falls through to the catch-all, which is exactly the alert you cannot afford to lose.

# alertmanager.yml
global:
  resolve_timeout: 5m

route:
  receiver: 'sink-catchall'
  group_by: ['alertname', 'cluster']
  group_wait: 30s
  group_interval: 1m
  repeat_interval: 1h
  routes:
    - matchers:
        - severity = critical
      receiver: 'sink-page'
      group_by: ['alertname', 'cluster']
      group_wait: 30s
      group_interval: 1m
      repeat_interval: 30m
      routes:
        - matchers:
            - team = dba
          receiver: 'sink-page-dba'
        - matchers:
            - team = payments-new
          receiver: 'sink-page-payments'
    - matchers:
        - severity = warning
      receiver: 'sink-warning'

receivers:
  - name: 'sink-catchall'
    webhook_configs:
      - url: 'http://sink:8080/'
        send_resolved: true
        timeout: 5s
  - name: 'sink-page'
    webhook_configs:
      - url: 'http://sink:8080/'
        send_resolved: true
        timeout: 5s
  - name: 'sink-page-dba'
    webhook_configs:
      - url: 'http://sink:8080/'
        send_resolved: true
        timeout: 5s
  - name: 'sink-page-payments'
    webhook_configs:
      - url: 'http://sink:8080/'
        send_resolved: true
        timeout: 5s
  - name: 'sink-warning'
    webhook_configs:
      - url: 'http://sink:8080/'
        send_resolved: true
        timeout: 5s

Five receivers, one URL. The webhook payload carries the receiver name, so the sink can tell you which route won without five separate services.

group_interval: 1m is shorter than the 5m you would run in production. It is short here because the lab has to observe a second notification for an evolving group inside a session; everything else is left at values you would actually deploy.

Run the static check:

$ amtool_file check-config alertmanager.yml

Checking 'alertmanager.yml'  SUCCESS
Found:
- global config
- route
- 0 inhibit rules
- 5 receivers
- 0 templates

SUCCESS means every receiver a route names exists and the YAML is well formed. It does not mean any alert you actually emit will reach the route you wrote it for.

Task 3: Prove the routing before you deploy it

amtool config routes test takes a label set and reports the receiver it would select and the group it would join. It reads the file; the server does not need to be running. Work through six label sets and record each answer in routing-table.md as you go — that table is the artefact you will re-run against this config every time somebody edits it.

for labels in \
  "alertname=ClusterDown severity=critical team=dba cluster=prod-eu-1" \
  "alertname=HostDown severity=critical team=dba cluster=prod-eu-1" \
  "alertname=HostDown severity=critical team=infra cluster=prod-eu-1" \
  "alertname=DiskFillingSoon severity=warning team=dba cluster=prod-eu-1" \
  "alertname=CheckoutErrors severity=critical team=payments cluster=prod-eu-1" \
  "alertname=CertExpiringSoon severity=info team=infra cluster=prod-eu-1"
do
  echo "--- $labels"
  # shellcheck disable=SC2086
  amtool_file config routes test --config.file=alertmanager.yml $labels
done

Read the six answers against what you intended:

Row five is the defect. The tree has a team = payments-new route, written during a re-org that renamed the team, but the alert rules emit team = payments. The route matches nothing. The alert is not lost — the parent severity = critical route catches it, so it still pages — but it pages the generic rota rather than the payments rota, and no error was raised anywhere. Fix the matcher:

        - matchers:
            - team = payments
          receiver: 'sink-page-payments'

Re-run the loop and confirm row five now selects sink-page-payments.

Row six is a design question rather than a bug. An info alert falling through to the catch-all is fine if somebody reads the catch-all, and a slow leak if nobody does. Decide which, and write the answer down.

Task 4: Start the stack and watch grouping

The sink, in full. It is deliberately small: it accepts a POST, prints what matters, and answers 200 so Alertmanager does not retry.

#!/usr/bin/env python3
"""Minimal Alertmanager webhook sink: log every notification to stdout."""
import json
from http.server import BaseHTTPRequestHandler, HTTPServer


class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        length = int(self.headers.get("Content-Length", 0))
        payload = json.loads(self.rfile.read(length) or b"{}")
        alerts = payload.get("alerts", [])
        print(json.dumps({
            "receiver": payload.get("receiver"),
            "status": payload.get("status"),
            "groupLabels": payload.get("groupLabels"),
            "count": len(alerts),
            "alertnames": sorted(a["labels"].get("alertname", "?") for a in alerts),
        }), flush=True)
        self.send_response(200)
        self.end_headers()

    def log_message(self, *args):
        return


HTTPServer(("0.0.0.0", 8080), Handler).serve_forever()

# docker-compose.yml
services:
  alertmanager:
    image: prom/alertmanager:v0.28.1
    command:
      - --config.file=/etc/alertmanager/alertmanager.yml
      - --storage.path=/alertmanager
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
      - am-data:/alertmanager
    ports:
      - "9093:9093"

  sink:
    image: python:3.12-alpine
    command: ["python3", "/srv/sink.py"]
    volumes:
      - ./sink.py:/srv/sink.py:ro

volumes:
  am-data:

$ docker compose up -d && curl -s http://localhost:9093/-/healthy && echo ' healthy'

In a second terminal, follow the sink:

docker compose logs -f sink

Now fire the incident from the scenario: one cluster-level alert and two host-level alerts, all critical, all team=dba, all in prod-eu-1. Send them together so they land inside the same group_wait:

amtool_live alert add alertname=ClusterDown severity=critical team=dba cluster=prod-eu-1
amtool_live alert add alertname=HostDown severity=critical team=dba cluster=prod-eu-1 instance=db-1
amtool_live alert add alertname=HostDown severity=critical team=dba cluster=prod-eu-1 instance=db-2

Wait 30 seconds — the group_wait — and read the sink.

$ docker compose logs sink | tail -4

{"receiver": "sink-page-dba", "status": "firing", "groupLabels": {"alertname": "ClusterDown", "cluster": "prod-eu-1"}, "count": 1, "alertnames": ["ClusterDown"]}
{"receiver": "sink-page-dba", "status": "firing", "groupLabels": {"alertname": "HostDown", "cluster": "prod-eu-1"}, "count": 2, "alertnames": ["HostDown", "HostDown"]}

Two notifications for three alerts. group_by: ['alertname', 'cluster'] put the two HostDown alerts in one group, because they agree on both labels, and left ClusterDown in a group of its own because its alertname differs. The group key is visible in groupLabels.

Note what the two hosts’ instance labels did not do: they are carried on the individual alerts inside the notification, and they had no effect on how many notifications were sent, because instance is not in group_by.

Task 5: Change the group key and watch the count change

Edit group_by on the severity = critical route to include instance:

      group_by: ['alertname', 'cluster', 'instance']

Reload without restarting. Alertmanager reloads routes, receivers, inhibit rules and templates in place; the silence store and the notification log are untouched.

$ amtool_file check-config alertmanager.yml && curl -sf -X POST http://localhost:9093/-/reload && echo reloaded

Confirm the running instance took the change rather than assuming it:

amtool_live config show | head -20

Alerts added with amtool alert add and no explicit end time expire after resolve_timeout — five minutes here. If more than five minutes have passed, re-send the three alerts before continuing. Then send them again and wait out group_wait:

amtool_live alert add alertname=HostDown severity=critical team=dba cluster=prod-eu-1 instance=db-1
amtool_live alert add alertname=HostDown severity=critical team=dba cluster=prod-eu-1 instance=db-2
amtool_live alert add alertname=HostDown severity=critical team=dba cluster=prod-eu-1 instance=db-3

The sink now logs three notifications where it logged one, each with count: 1 and an instance in groupLabels. Three hosts in one incident became three pages.

Put group_by back to ['alertname', 'cluster'] and reload again before continuing.

Task 6: Add an inhibit rule, then find its edge

The ClusterDown page tells the whole story. The HostDown pages are consequences. Append to alertmanager.yml:

inhibit_rules:
  - source_matchers:
      - alertname = ClusterDown
    target_matchers:
      - alertname =~ "HostDown|ServiceDown"
    equal: ['cluster']

Check and reload, then re-send the incident: ClusterDown and two HostDown alerts in prod-eu-1.

amtool_file check-config alertmanager.yml
curl -sf -X POST http://localhost:9093/-/reload && echo reloaded
amtool_live alert add alertname=ClusterDown severity=critical team=dba cluster=prod-eu-1
amtool_live alert add alertname=HostDown severity=critical team=dba cluster=prod-eu-1 instance=db-1
amtool_live alert add alertname=HostDown severity=critical team=dba cluster=prod-eu-1 instance=db-2

Ask Alertmanager what state each alert is in:

$ curl -s http://localhost:9093/api/v2/alerts | jq -r '.[] | "\(.labels.alertname) \(.labels.cluster) state=\(.status.state) inhibitedBy=\(.status.inhibitedBy | length)"'

ClusterDown prod-eu-1 state=active inhibitedBy=0
HostDown prod-eu-1 state=suppressed inhibitedBy=1
HostDown prod-eu-1 state=suppressed inhibitedBy=1

The two HostDown alerts are suppressed, not gone. They are in the alert store, they show in the UI, and they will be in the post-incident timeline. They simply do not generate a notification while the source is firing — and the sink confirms it, with one notification for ClusterDown and none for HostDown.

Now the part that matters. Fire a HostDown in a different cluster, where no ClusterDown is firing:

amtool_live alert add alertname=HostDown severity=critical team=dba cluster=prod-us-2 instance=db-9

It is active, not suppressed, and it reaches the sink. That is equal: ['cluster'] doing its job: the rule only suppresses where the source and the target are causally related.

Task 7: Silence one host without silencing the fleet

A planned reboot of db-1 is scheduled. The intent is to mute alerts for that host, for that window, and nothing else.

amtool_live silence add \
  alertname=HostDown cluster=prod-eu-1 instance=db-1 \
  --duration=15m \
  --author="you@example.com" \
  --comment="CHG-1042 planned reboot of db-1; expires with the window"

Confirm it, and confirm what it covers:

amtool_live silence query
amtool_live alert add alertname=HostDown severity=critical team=dba cluster=prod-eu-1 instance=db-1
amtool_live alert add alertname=HostDown severity=critical team=dba cluster=prod-eu-1 instance=db-7

$ curl -s http://localhost:9093/api/v2/alerts | jq -r '.[] | select(.labels.alertname == "HostDown") | "\(.labels.instance) state=\(.status.state) silencedBy=\(.status.silencedBy | length)"'

db-1 state=suppressed silencedBy=1
db-7 state=active silencedBy=0

Now do the thing everybody does at 03:00 and see what it costs. Add a second silence with one matcher:

BROAD_SILENCE=$(amtool_live silence add alertname=HostDown \
  --duration=15m \
  --author="you@example.com" \
  --comment="quick mute, will scope it properly later")
echo "created $BROAD_SILENCE"

amtool silence add prints the id of the silence it created, which is why it is worth capturing rather than reading back out of a list — the next command has to expire exactly this one and not the scoped one.

Re-query the alerts. Every HostDown in the estate is now suppressed, including db-7, including clusters you have never heard of, for fifteen minutes. This is the silence that swallows the real incident, and the only thing separating it from the correct one above is two matchers.

Expire it and confirm the blast radius shrank back:

amtool_live silence expire "$BROAD_SILENCE"
amtool_live silence query

Confirm db-7 is active again, and that the scoped silence on db-1 is untouched.

Validation

1. amtool_file check-config alertmanager.yml exits 0 and reports five receivers and one inhibit rule.
2. Your routing table has six rows, and rerunning the Task 3 loop reproduces every receiver in it — including sink-page-payments for the payments row.
3. Reverting team = payments to team = payments-new makes that row select sink-page again. Try it, then revert.
4. The sink log contains one notification with count: 2 for the grouped HostDown pair, and three notifications with count: 1 from the instance-grouped run.
5. With the inhibit rule loaded, HostDown in prod-eu-1 is suppressed while ClusterDown fires there, and HostDown in prod-us-2 is active at the same moment.
6. The scoped silence suppresses db-1 and not db-7; the unscoped one suppresses both; expiring it restores db-7.
7. curl -s http://localhost:9093/api/v2/status | jq '.cluster.status' reports a ready single-node cluster.

Expected Outcome

obs-alertmanager-lab/
├── alertmanager.yml
├── docker-compose.yml
├── routing-table.md
└── sink.py

An alertmanager.yml with a severity-first tree, five receivers, one equal:-scoped inhibit rule, and no route that matches nothing. A routing-table.md recording six label sets and the receiver each selects. Captured sink output showing the same three alerts producing one notification or three, depending only on group_by. One expired silence and one still-scoped silence in the store.

Troubleshooting

amtool_live fails with “container is not running”. The helper uses docker compose exec, so the stack has to be up and you have to be in the lab directory. docker compose ps first.

check-config fails with an undefined receiver. A route names a receiver that is not in receivers:. The message names it. This is the one class of routing mistake the static check does catch.

Nothing appears in the sink. Check the order of the three things that have to be true: the alert exists (curl -s localhost:9093/api/v2/alerts), it is active rather than suppressed, and group_wait has elapsed. If all three hold, look at the Alertmanager log — a webhook that cannot be reached logs a dial error naming the URL.

The sink container exits immediately. The bind mount put a directory where sink.py should be, which happens when Compose is started before the file exists. docker compose down, confirm sink.py is a file, start again.

An alert vanishes between commands. Alerts posted without an end time expire after resolve_timeout, five minutes by default. This is correct behaviour — Prometheus re-sends firing alerts continuously, and an alert that stops being re-sent is an alert that stopped firing — but it means a lab step that takes six minutes needs its alerts re-sent.

A reload appears to do nothing. POST /-/reload returns 200 whether or not the file on disk is what you think it is. amtool_live config show prints what the running process actually parsed; trust that over the editor.

Cleanup

Two containers, one named volume, one network, one directory.

$ cd "$HOME/obs-alertmanager-lab" && docker compose down -v

Confirm the port is free again, matching what you recorded in Task 1:

docker compose ps
ss -ltnp 2>/dev/null | grep -E ':9093\b' || echo 'port free'

$ mkdir -p "$HOME/obs-lab-deliverables" && cp -a "$HOME/obs-alertmanager-lab/alertmanager.yml" "$HOME/obs-lab-deliverables/alertmanager-lab.yml" && rm -rf "$HOME/obs-alertmanager-lab"

The two images stay in the local cache. Remove them if you are done with the Alertmanager labs:

docker image rm prom/alertmanager:v0.28.1 python:3.12-alpine

Production notes

• The config is code and belongs in Git. Alertmanager’s durable state is the file plus the silence store; the file is the half you can version. A pull request that changes routing should carry the label sets it changes the answer for.
• Two gates, in order, in CI. amtool check-config on the file, then amtool config routes test for every rota. The second is the one that catches a re-org.
• Reload, do not restart. A SIGHUP or POST /-/reload swaps routes, receivers, inhibit rules and templates while leaving the silence store and notification log alone. A restart drops the in-memory index and reloads it from disk, which is usually fine and is not always: the maintenance silence somebody created ten minutes ago is the thing you will miss.
• Keep the previous file. Copy alertmanager.yml to alertmanager.yml.bak before every change. Rolling back a routing mistake at 03:00 should be one move, not an editing session.
• Secrets never go in the file. Every integration takes a _file variant — api_url_file, routing_key_file, auth_password_file — and the file is mounted 0600. The YAML is readable by everyone with read access to the repository; the webhook URL that pages your entire company should not be.
• Set a timeout: on every webhook. The default is ten seconds and an upstream that accepts the connection and hangs will consume all of it, per notification. Five seconds is more honest for a service you do not control.

What You Learned

• Severity before team. The first split guarantees a critical alert pages somebody; splitting on team first means an unknown team value is an alert nobody receives.
• check-config and routes test answer different questions. The first asks whether the file loads. The second asks whether your alerts reach the rota you wrote the route for, and only for the label sets you hand it.
• A route that matches nothing is silent. The payments-new route raised no error anywhere; the catch-all absorbed the alerts and the payments rota simply never heard from them.
• group_by decides the page count. The same three alerts produced one notification or three, and nothing about the alerts changed.
• Inhibition suppresses notifications, not alerts. The HostDown alerts stayed in the store as suppressed and remain available for the timeline.
• equal: is the scope, and its absence is the bug. With equal: ['cluster'] the rule suppressed inside one cluster and not across the estate; that clause is the whole difference between a useful rule and a fleet-wide mute.
• A silence is a scalpel or a blackout. Two extra matchers were the difference between muting one host for a change window and muting every HostDown everywhere for fifteen minutes.