Alertmanager HA · Observability

Two Alertmanagers. A network blip in the middle of a paging incident. One node loses contact with the other for 90 seconds. Both nodes think they are the only cluster member. Both nodes process the same batch of incoming alerts. Both nodes fire the same notifications to the same on-call channel. The on-call gets paged twice for the same condition.

Alertmanager HA is built on a gossip protocol. The protocol is eventually consistent: every node eventually agrees on the state of the cluster, but during partitions the two sides disagree. The disagreement is acceptable for some failure shapes (silences are eventually consistent, so a silence created on node A reaches node B after a few seconds) and dangerous for others (notifications are processed per node, so two nodes can fire two pages).

The discipline is to know which is which.

What it is

Alertmanager HA is the practice of running two or more Alertmanager instances that share the same alert pipeline. The cluster is gossip-based: nodes discover each other through a memberlist protocol (the same one HashiCorp Consul uses for its cluster membership). There is no leader election, no primary/secondary split, no shared state service. Every node has the same view of the alert pipeline within a few seconds of any change.

The contrast is single-instance Alertmanager. A single Alertmanager is a single point of failure. If the host dies, the on-call stops receiving pages until the process is restarted or replaced. The fix is to run two or three. The trade-off is that two or three introduce their own failure shapes (gossip partitions, duplicate notifications), and the discipline is to handle those deliberately.

Why a sysadmin cares

Alertmanager is the on-call’s last line of defence. If Alertmanager is down, the team does not know it is down. If Alertmanager is misconfigured, the team gets paged twice per incident and learns to ignore pages.

Two failure shapes appear repeatedly in production Alertmanager deployments:

The single-node that died. The cluster was a single Alertmanager. The host OOMed during a paging storm. The on-call learned about the production outage from a user complaint rather than from an alert. The fix is to run at least two Alertmanagers.
The partitioned cluster. Two Alertmanagers behind a load balancer with a flaky link between them. The link flaps. During the flap, both nodes process notifications independently. Pages fire twice. The on-call learns to ignore the second page and misses a real incident. The fix is to know the partition behaviour and to apply cluster.replica-label so duplicate-scrape alerts collapse before notification.

How it works

The Alertmanager cluster gossip:

   +------------+      gossip over UDP/TCP       +------------+
   | Alertmgr A | <---------------------------> | Alertmgr B |
   | peer list: |                                | peer list: |
   |  A, B, C   |                                |  A, B, C   |
   +-----+------+      gossip over UDP/TCP       +-----+------+
         |     <--------------------------->            |
         |                                            |
         +----------------+---------------------------+
                          |
                          v
                  +----------------+
                  | Alertmgr C     |
                  | peer list:     |
                  |  A, B, C       |
                  +-------+--------+
                          |
                          v
                Silences / Notifications
                deduplicated per alert

Every node runs the memberlist protocol. Every node periodically sends a ping to a peer; if the peer does not respond within a few seconds, the node is marked as failed and the failure is gossiped to the rest of the cluster. The membership view is eventually consistent across the cluster within a few seconds.

The notification path:

Prometheus sends an alert to the Alertmanager endpoint (HA VIP, DNS round-robin, or a load balancer).
The receiving node processes the alert. If the node is part of a cluster, it gossip-replicates the alert state to every peer.
After the inhibition and silence phases, the node decides whether to notify. The decision is per node.
The node fires the notification through the configured receiver (Slack, PagerDuty, Opsgenie, email).

The deduplication happens at the notification step (step 3). The cluster does not elect a leader for notification; every node decides for itself. If the cluster has replica-label configured, the decision accounts for the replica label and two alerts with the same name but different replica values are treated as the same alert. If the flag is missing, they are treated as distinct alerts and both nodes notify.

Under the hood

The gossip layer is hashicorp/memberlist. The library is the same one Consul uses for its cluster membership. The protocol uses UDP for ping/ack and TCP for state push (the heavier “full state sync” messages). Every node maintains a peers list; the list is updated on every ping/ack/failure event.

The failure detection in memberlist is indirect. Node A suspects node B after B fails to ack a number of pings (the suspicion threshold, configurable via --cluster.suspicion-multiplier). The suspicion is gossiped across the cluster. If a quorum of nodes also suspects B, B is declared failed and removed from the peer list. The whole cycle takes seconds, not minutes, but it is not instantaneous.

The alert pipeline is replicated via gossip. When a node receives a new alert, it broadcasts the alert state to every peer. The peers update their local view. After a brief delay (the gossip round-trip), every node has the same alert state. Silences are replicated the same way. The replication is eventually consistent: a silence created on node A reaches node B within a few seconds.

The notification step is not gossip-replicated. The notification path is local to each node. Each node evaluates the alert pipeline independently and decides whether to notify. The cluster.replica-label setting is the synchronisation point: when the label is set, the decision accounts for the replica value, so two copies of the same alert (one with replica=a, one with replica=b) are treated as the same alert at notification time.

The gossip protocol requires direct connectivity between every node pair, or at least between every node and a small set of seeds (--cluster.peer). A node that loses connectivity to every peer (the seed list is empty from its perspective) declares itself a single-node cluster. From that node’s view, it is the only Alertmanager. If the partition heals, the two sides re-merge, but during the partition both sides processed notifications independently. The on-call may have received double pages during the window.

The --cluster.listen-address flag binds the gossip UDP/TCP ports. The default is :9094. The --cluster.peer flag lists one or more seed nodes. Every node needs to be able to reach every seed. Seeds are not the entire cluster; they are the nodes to contact to learn the full peer list. In a three-node cluster, each node lists the other two as seeds.

How to configure it

The minimum Alertmanager HA cluster is two nodes. The production-typical cluster is three.

# /etc/alertmanager/alertmanager.yml (on every node)
global:
  resolve_timeout: 5m

route:
  receiver: 'default'
  group_by: ['alertname', 'cluster']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - severity = critical
      receiver: 'pager'
      group_wait: 10s

receivers:
  - name: 'default'
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/...'
        channel: '#alerts'

  - name: 'pager'
    pagerduty_configs:
      - service_key: '<key>'

cluster:
  listen-address: ""                  # empty -> bind 0.0.0.0:9094
  replica-label: replica              # dedup across duplicate-scrape pairs
  peers:
    - am-a.prod.example.com:9094
    - am-b.prod.example.com:9094
    - am-c.prod.example.com:9094

inhibit_rules:
  - source_matchers:
      - severity = critical
    target_matchers:
      - severity = warning
    equal: ['alertname', 'cluster']

The cluster.listen-address is empty so Alertmanager binds all interfaces on port 9094. The cluster.peer list is the same on every node; it tells every node who to contact for membership. The cluster.replica-label is the dedup key.

Validate before starting:

# SEVERITY: READ-ONLY
amtool check-config /etc/alertmanager/alertmanager.yml

Start each node:

# SEVERITY: SERVICE-IMPACT
alertmanager \
  --config.file=/etc/alertmanager/alertmanager.yml \
  --storage.path=/var/lib/alertmanager \
  --web.listen-address=0.0.0.0:9093 \
  --cluster.listen-address=0.0.0.0:9094 \
  --cluster.peer=am-a.prod.example.com:9094 \
  --cluster.peer=am-b.prod.example.com:9094 \
  --cluster.peer=am-c.prod.example.com:9094

On each node, the cluster.peer list is identical. The node contacts the listed peers, learns the full peer list, and gossips with everyone.

The Prometheus side needs to send alerts to the cluster, not to a single node. The standard pattern is a DNS A record with all three nodes, or a load balancer in front of the three nodes. The Prometheus configuration:

# /etc/prometheus/prometheus.yml
alerting:
  alertmanagers:
    - static_configs:
        - targets: ['am-vip.prod.example.com:9093']

If the VIP or DNS is unhealthy, every Prometheus must fall back to a different target, otherwise the alert path is lost.

How to validate it

The first check is that every node sees every other node in its peer list:

# SEVERITY: READ-ONLY
curl -s http://am-a:9093/api/v1/status | jq '.cluster.status'

Expected output (illustrative):

{
  "status": "ready",
  "peerStatus": {
    "am-a.prod.example.com:9094": { "name": "am-a", "address": "10.0.1.10:9094", "healthy": true },
    "am-b.prod.example.com:9094": { "name": "am-b", "address": "10.0.1.11:9094", "healthy": true },
    "am-c.prod.example.com:9094": { "name": "am-c", "address": "10.0.1.12:9094", "healthy": true }
  }
}

Three healthy peers. If any peer is missing or unhealthy, the gossip has not converged yet, or the node cannot reach the peer.

The second check is that the alert pipeline is identical on every node:

# SEVERITY: READ-ONLY
amtool config show --alertmanager.url=http://am-a:9093 | head -30
amtool config show --alertmanager.url=http://am-b:9093 | head -30
amtool config show --alertmanager.url=http://am-c:9093 | head -30

The outputs should be byte-for-byte identical. A drift in the configuration between nodes causes one node to evaluate a different route tree from the others and fire different notifications.

The third check is that the replica-label is honoured. Trigger an alert from a Prometheus with replica=a, then the same alert from replica=b. Inspect the active alerts:

# SEVERITY: READ-ONLY
amtool alert query --alertmanager.url=http://am-a:9093

Expected output (illustrative): one alert, with replica=a in the labels. The replica b copy of the alert should not appear as a separate alert. If both appear, the cluster.replica-label flag is missing or misconfigured.

The fourth check is that notifications are not doubled. Send the same alert from both Prometheuses and watch the receiver:

# SEVERITY: CONFIGURATION
amtool alert query --alertmanager.url=http://am-a:9093 \
  | grep -c 'integrations'

The expected count is one notification per alert (one notification to Slack or PagerDuty), not two.

How it can fail

Six failure modes appear repeatedly in production Alertmanager clusters.

Single-node deployment. The HA target was two nodes but only one was started because the second was “still being deployed.” Symptom: the single node processes every alert; when it dies, no pages are sent. The fix is to verify every node is running with curl /api/v1/status.
Partitioned cluster firing duplicates. A three-node cluster loses connectivity between two halves (one node alone, two nodes together). Symptom: during the partition, both halves process notifications independently, and pages fire twice. The fix is cluster.replica-label and a tolerance for the brief window where dedup has not caught up.
The replica label is not set on Alertmanager. The Prometheuses have replica=a and replica=b external labels. The Alertmanager cluster has no replica-label configured. Symptom: every alert fires twice. The fix is to add cluster.replica-label: replica to every node and to reload.
The cluster.peers list is empty or stale. A new node is added with no cluster.peer directives. Symptom: the new node cannot discover the existing cluster; it runs as a single-node cluster; alerts sent to it are not gossiped. The fix is to populate cluster.peer with at least one reachable existing node.
DNS resolves the VIP to a dead node. The Prometheus alerting target is a DNS name that resolves to a single Alertmanager that has crashed. Symptom: no alerts reach any Alertmanager. The fix is to ensure the DNS returns multiple addresses (round-robin) or to use a load balancer that performs health checks.
The gossip port is firewalled. The cluster listen-address is bound, but the firewall on the host blocks UDP/TCP 9094 between nodes. Symptom: nodes never see each other; each runs as a single-node cluster; alerts fire N times for an N-node cluster. The fix is to open the port in the host firewall and the network policy.

How to troubleshoot it

The diagnostic order when pages are doubled or not arriving:

Confirm the cluster membership. curl /api/v1/status on each node; compare the peer lists.
Confirm the replica-label setting. amtool config show on each node; check the cluster block.
Confirm the Prometheus alerting target resolves to a healthy node. dig or nslookup on the VIP and curl on each IP.
Confirm the gossip port is open between nodes. nc -zvu am-b 9094 and nc -zv am-b 9094.
Confirm the alertmanager configuration is identical across nodes. amtool config show diff.
Confirm the notification configuration is identical and the receivers are reachable.

What happens if the network between two halves of a three-node Alertmanager cluster partitions?

A network split isolates one node from the other two. The isolated node declares itself a single-node cluster (it cannot reach any peer). The other two nodes gossip normally and form a two-node cluster. Both halves process alerts independently during the partition.

If the Prometheuses are also split (for example, by the same network failure), each half of the Alertmanager cluster receives alerts only from the Prometheuses it can reach. Pages fire once per half. No duplication. This is the benign case.

If the Prometheuses can reach both halves of the Alertmanager cluster (a separate failure shape), both halves receive the same alert and both fire notifications. Pages fire twice for the same condition. The on-call may see two pages in a row for the same incident. If the partition lasts longer than repeat_interval, the pages keep firing on each half on its own schedule.

The fix is cluster.replica-label plus a tolerance window. The label collapses the two copies before notification, so the duplication is contained. The tolerance window is the gossip convergence time (typically 5-10 seconds); during that window, the two halves may briefly diverge on whether the alert is already firing. After the window, both halves see the same state and stop duplicating.

Security implications

Alertmanager exposes the web UI on port 9093 and the API on the same port. The API allows silencing, alert acknowledgement, and configuration inspection. Authentication is via basic auth or OAuth2 proxy. A misconfigured reverse proxy without auth exposes the silencer to anyone who can reach the port.

The gossip port (9094) does not require authentication, but the protocol is internal-cluster only. The port should be firewalled to allow traffic only from the cluster’s node IPs. Exposing 9094 to the public internet allows any host to join the cluster and inspect alert state.

The notification endpoints (Slack webhooks, PagerDuty keys, Opsgenie tokens) are credentials. The configuration file should be readable only by the Alertmanager user, and the secrets should be sourced from a secret store rather than committed to the repository.

Performance implications

Alertmanager is a low-throughput service compared to Prometheus. The alert evaluation runs every 15-30 seconds; each evaluation produces a small number of alert state changes; notifications are typically a few per hour. A single Alertmanager can handle the alert pipeline for tens of thousands of series.

The gossip overhead scales with the cluster size. Each node gossips with every other node; the message rate is proportional to O(N * frequency). For a 3-node cluster, the overhead is small. For a 20-node cluster (unusual), the overhead becomes significant and the cluster gossip can fall behind under load.

The notification path is synchronous per receiver. A slow receiver (a Slack webhook that times out, for example) blocks the notification pipeline until the timeout. The default notification timeout is configurable; the production value is typically 30 seconds per receiver.

Production guidance

Run three Alertmanager nodes. Two is acceptable for small deployments; three gives tolerance for one node failure plus one network partition.
Place the nodes in different availability zones or failure domains. A zone failure should not take down the whole cluster.
Always set cluster.replica-label: replica on every node. The flag is the dedup point; without it, duplicate-scrape Prometheuses cause double pages.
Use a DNS round-robin or load balancer in front of the Alertmanager cluster for the Prometheus alerting target. The DNS must return multiple addresses and the addresses must be healthy.
Open the gossip port (9094) between cluster nodes in the firewall and the network policy.
Keep the Alertmanager configuration in a version-controlled repository and reload every node from the same source. Drift between nodes causes divergent notification paths.
Monitor the cluster membership. Alert on alertmanager_cluster_members dropping below the expected count.
Monitor the notification path. Alert on alertmanager_notifications_failed_total rising.

Verification

You should now be able to answer:

How does Alertmanager gossip propagate the alert state across the cluster, and how does it decide which node fires a notification?
What is the right number of Alertmanager nodes for a production HA deployment, and why is two acceptable but three typical?
Which flag tells Alertmanager to dedup alerts across duplicate-scrape Prometheuses, and on which config block does it live?
What is the most common Alertmanager HA failure shape, and how does it surface?
How does a partitioned cluster process notifications, and what limits the duplication?