Redundant Alertmanager

Why this matters in production

Alertmanager is where alerts become notifications. A single instance failing means alerts fire correctly and nobody hears them.

The cluster

alertmanager \
  --config.file=/etc/alertmanager/alertmanager.yml \
  --cluster.listen-address=0.0.0.0:9094 \
  --cluster.peer=alertmanager-02:9094 \
  --cluster.peer=alertmanager-03:9094

Each instance lists the others as peers. They form a gossip cluster and coordinate rather than acting independently.

amtool cluster show --alertmanager.url=http://alertmanager-01:9093
curl -s http://alertmanager-01:9093/api/v2/status | python3 -c '
import sys,json; d=json.load(sys.stdin)
print("name:", d["cluster"]["name"], "status:", d["cluster"]["status"])
print("peers:", [p["address"] for p in d["cluster"]["peers"]])'

What the gossip protocol synchronises

The notification log is the important one: it prevents every instance sending the same notification. Instances coordinate so one sends and the others record that it was sent.

# a silence created on one instance appears on all
amtool silence add alertname=CephNearfull --duration=4h \
  --alertmanager.url=http://alertmanager-01:9093
amtool silence query --alertmanager.url=http://alertmanager-02:9093

Verifying membership

for a in alertmanager-01 alertmanager-02 alertmanager-03; do
  printf '%-18s ' "$a"
  curl -s "http://$a:9093/api/v2/status" | python3 -c '
import sys,json; d=json.load(sys.stdin)
print(d["cluster"]["status"], len(d["cluster"]["peers"]), "peers")' 2>/dev/null     || echo unreachable
done

A cluster where instances cannot see each other is worse than a single instance: each acts independently and every notification is sent multiple times.

- alert: AlertmanagerClusterDegraded
  expr: alertmanager_cluster_members < 3
  for: 10m
  labels: { severity: ticket }

The remaining failure modes

Configuration drift between instances is the subtle one: each applies its own routing, so an alert may be routed differently depending on which instance handles it.

# verify configuration is identical
for a in alertmanager-01 alertmanager-02; do
  curl -s "http://$a:9093/api/v2/status" | python3 -c '
import sys,json,hashlib
print(hashlib.sha256(json.load(sys.stdin)["config"]["original"].encode()).hexdigest()[:16])'
done

Quiz

Production discipline

Monitor alertmanager_cluster_members — partitioned instances each appear healthy while every notification is sent once per instance. Verify configuration is identical across instances; gossip does not synchronise it, and drift produces routing that depends on which instance handles the alert.

Cross-course references

• Kubernetes: gossip-based coordination appears in many controllers for the same reason
• Linux: split-brain in any clustered service produces duplicated rather than absent work