Alertmanager Delivery

A page fires on the platform team at 09:00 for a payment service latency spike. The Alertmanager UI shows the alert as firing and resolved within the same minute. The on-call engineer receives nothing. The webhook integration on the receiver was deleted by a security audit two weeks earlier. The receiver configuration reload was attempted; the config_last_reload_success_timestamp_seconds shows a stale timestamp. Nobody notices because no alert watches the notification failure counter.

Alertmanager delivery is the part of the platform most likely to fail silently. The metric counter is right there. The alert that watches the counter is the missing piece.

What it is

Alertmanager receives alerts from Prometheus (or other sources), groups them, applies routes and inhibitions, and delivers the result to receivers (PagerDuty, Slack, webhooks, email, etc.). The metrics it exposes on /metrics (port 9093 by default) include:

• alertmanager_notifications_total{integration} — counter of notifications attempted per integration.
• alertmanager_notifications_failed_total\{integration, reason\} — counter of notifications that failed, by integration and reason (the reason label distinguishes config errors, client errors, and server errors).
• alertmanager_http_request_duration_seconds — histogram of HTTP latency to upstream integrations.
• alertmanager_cluster_health_score{} — gauge. Value between 0 and 1; drops when peers are unreachable.
• alertmanager_cluster_peers — gauge. Number of configured peers.
• alertmanager_config_last_reload_success_timestamp _seconds — gauge. Wall-clock timestamp of the last successful config reload.
• alertmanager_silences — gauge. Number of active silences.
• alertmanager_alerts — gauge. Number of active alerts.
• alertmanager_dispatcher_aggregation_group_limit_reached _total — counter. Times the dispatcher dropped an alert because a group exceeded the limit.

The right approach is to alert on:

1. rate(alertmanager_notifications_failed_total[5m]) > 0 — delivery is failing.
2. alertmanager_cluster_health_score < 0.5 — cluster split.
3. time() - alertmanager_config_last_reload_success_timestamp _seconds > 600 — config did not reload.

Why a sysadmin cares

An alert that fires but does not deliver is the same as no alert. The cost is paid by the user: the outage runs longer because the on-call engineer is asleep at the controls. The single most common production shape is a webhook URL that has been rotated, an SMTP relay that has been changed, or a Slack OAuth token that has been revoked. The fix is five lines of YAML in the right receiver; the diagnosis is the metric counter that already exists.

A second reason: Alertmanager is a stateful cluster. Splits and silences are silent. A misconfigured silence that hides a critical alert is a worse failure than a noisy alert that pages by mistake.

How it works

Prometheus
   |
   v
Alertmanager (n peers, gossip on 9094)
   |
   v
+----------------------+
| dispatcher           |   groups alerts by label set
|  - group by labels   |   waits group_wait (default 30s)
|  - group_wait        |   then routes
|  - group_interval    |
|  - repeat_interval   |
+----------------------+
   |
   v
+----------------------+
| route tree           |   matchers -> child routes
|  - receivers         |   -> receivers (Slack, PD, etc.)
|  - inhibit_rules     |   -> inhibit / silence
|  - mute_time_intervals
+----------------------+
   |
   v
+----------------+----------------+
|                |                |
v                v                v
receiver        receiver        receiver
(webhook)       (slack)         (pagerduty)

The dispatcher groups alerts by a configurable label set (group_by), waits group_wait for additional alerts in the same group, then sends the group to the matched route. Each route ends in a receiver. The receiver attempts delivery to one or more integrations (Slack channel, PagerDuty service, webhook URL, etc.). A failure increments alertmanager_notifications_failed_total with a reason label distinguishing client (4xx), server (5xx), and config (misconfigured URL, missing auth).

The cluster is gossip-based. Each peer replicates the notification pipeline and the silences/inhibitions view. A peer that becomes unreachable drops the cluster health score.

Under the hood

How to configure it

Delivery-failure alerts

# /etc/observer/rules/alertmanager.yml
groups:
  - name: alertmanager-health
    rules:
      # 1. Any integration is failing delivery.
      - alert: AlertmanagerNotificationFailing
        expr: |
          rate(alertmanager_notifications_failed_total[5m]) > 0
        for: 5m
        labels:
          severity: critical
          team: platform
        annotations:
          summary: |
            Alertmanager delivery failing for
            {{ $labels.integration }} (reason:
            {{ $labels.reason }})

      # 2. Cluster health degraded.
      #    A drop below 0.5 means at least one peer is
      #    unreachable and the cluster is operating at
      #    reduced redundancy.
      - alert: AlertmanagerClusterDegraded
        expr: |
          alertmanager_cluster_health_score < 0.5
        for: 2m
        labels:
          severity: critical
          team: platform

      # 3. Cluster has fewer peers than configured.
      - alert: AlertmanagerClusterPeerMissing
        expr: |
          alertmanager_cluster_peers
            < on() group_left()
              (count by () (alertmanager_cluster_peers) > 0)
            # the right comparison is to the configured count;
            # this expression catches a peer disappearing.
        for: 5m
        labels:
          severity: warning
          team: platform

      # 4. Config did not reload successfully.
      - alert: AlertmanagerConfigReloadStale
        expr: |
          time() - alertmanager_config_last_reload_success_timestamp_seconds
            > 600
        for: 0m
        labels:
          severity: warning
          team: platform

      # 5. Dispatcher hit the aggregation group limit.
      #    Catches a flood that is being silently dropped.
      - alert: AlertmanagerDispatcherGroupLimitReached
        expr: |
          rate(alertmanager_dispatcher_aggregation_group_limit_reached_total[5m]) > 0
        for: 5m
        labels:
          severity: warning
          team: platform

      # 6. Active silences above a known ceiling.
      #    Catches the "silence sprawl" shape where every
      #    alert is silenced by accident.
      - alert: AlertmanagerSilencesExcessive
        expr: alertmanager_silences > 200
        for: 10m
        labels:
          severity: warning
          team: platform

Receiver health check via synthetic alert

A weekly synthetic alert that fires through every receiver catches delivery failures before a real incident does:

# /etc/prometheus/rules/canary-delivery.yml
groups:
  - name: delivery-canary
    rules:
      - alert: DeliveryCanaryWeekly
        expr: vector(1)
        for: 1m
        labels:
          severity: info
          canary: "true"
          receiver: "all"
        annotations:
          summary: "Weekly delivery canary - should reach every receiver"

This alert is routed to every receiver in the route tree. If any receiver fails to deliver, the counter alertmanager_notifications_failed_total{canary="true"} climbs. Inspect the reason label to find the broken integration.

How to validate it

Confirm Alertmanager is exposing delivery metrics:

# READ-ONLY
curl -s http://alertmanager.internal:9093/metrics \
  | grep -E '^alertmanager_notifications_(total|failed)_total' \
  | head -20

A healthy output shows one line per integration per label combination. The total counter increments on every notification; the failed counter is at zero on a healthy cluster.

Confirm the cluster is at full health:

# READ-ONLY
curl -s 'http://alertmanager.internal:9093/api/v2/status' \
  | jq '.cluster.status, .cluster.peers'

status: ready and peers matching the configured count is the healthy shape. status: settling means a peer is joining or leaving; wait one gossip cycle.

Confirm a canary delivery to one receiver end-to-end:

# READ-ONLY - send a test alert through amtool.
amtool alert-query alertmanager.internal:9093 \
  --alertmanager.url=http://alertmanager.internal:9093 \
  | head -5

# Trigger a synthetic alert.
curl -s -X POST 'http://alertmanager.internal:9093/api/v2/alerts' \
  -H 'Content-Type: application/json' \
  -d '[{
        "labels": {"alertname":"DeliveryCanary","severity":"info"},
        "annotations":{"summary":"manual delivery canary"}
      }]'

# Watch the counter.
watch -n 2 'curl -s http://alertmanager.internal:9093/metrics \
  | grep alertmanager_notifications_total'

A working setup shows the total counter incrementing on the receiver and the failed counter staying at zero. If the failed counter increments, the reason label names the failure.

How it can fail

1. Webhook URL rotated away

The receiver configuration still references a webhook URL that has been deleted. Symptom: alertmanager_notifications_failed_total\{integration= "webhook", reason="config"\} increments. Action: update the receiver URL; confirm alertmanager_config_last_reload_success_timestamp_seconds advances after the reload.

2. SMTP relay unreachable

The email receiver targets an SMTP server that has been moved or is firewalled. Symptom: alertmanager_notifications_failed_total\{integration= "email", reason="server"\} increments; the SMTP server’s own logs show connection refused from the Alertmanager host. Action: update the SMTP target or fix the firewall; confirm a manual test email works.

3. Slack OAuth token revoked

The Slack workspace owner revoked the OAuth token used by the Slack receiver. Symptom: alertmanager_notifications_failed_total\{integration= "slack", reason="client"\} increments; the AM log shows the 4xx from Slack. Action: regenerate the token and update the receiver configuration.

4. PagerDuty routing key wrong

The routing key was mistyped in a copy-paste. Symptom: alertmanager_notifications_failed_total\{integration= "pagerduty", reason="client"\} increments. Action: fix the key in the receiver config; test with a synthetic alert.

5. Cluster split

One of three AM peers is unreachable (network partition, OOM, host down). Symptom: alertmanager_cluster_health _score drops below 1.0; alerts delivered to the isolated peer are not gossiped. Action: restore the partition; if the peer is genuinely down, remove it from the cluster config to silence the health-score noise.

6. Silences hide a critical alert

A silence is in place that matches a critical alert’s labels. Symptom: alertmanager_silences is non-zero; the UI shows the silence; the alert never pages. Action: expire the silence or narrow its matchers; document the reason for the silence in the silence comment.

7. Inhibition rule too broad

An inhibit_rule matches a critical alert by mistake. Symptom: critical alerts are marked as inhibited; the counter alertmanager_dispatcher_aggregation_group_limit _reached_total may also climb. Action: narrow the target_matchers on the inhibit rule; test by firing the critical alert in staging.

8. Notification queue full

AM is delivering faster than the upstream integration can absorb. Symptom: alerts are queued internally; AM logs warnings about rate limits. Action: reduce rate_limit / burst if upstream can absorb more; otherwise increase the rate-limit tolerance on the upstream integration.

How to troubleshoot it

Security implications

The receiver configuration contains credentials for every upstream integration: Slack OAuth tokens, PagerDuty routing keys, SMTP passwords, webhook HMAC secrets. The AM configuration file is the secrets file. Store it in a secret manager; rotate on the same schedule as application credentials.

Webhook URLs can be weaponised as a data-exfiltration path. A misconfigured webhook can be pointed at an attacker- controlled endpoint that captures alert contents (which often include sensitive labels like user IDs, account IDs, request paths). Restrict webhook URLs to allow-listed domains via outbound firewall rules.

The AM API (/api/v2) requires authentication in production. The default is no auth; the production posture is basic auth, bearer token, or reverse-proxy auth. The same rule applies to the UI.

Performance implications

Alertmanager is bounded by the slowest integration. A single 5xx-returning webhook that retries for 30 seconds holds a dispatcher worker for that duration. The right sizing pattern is:

• Tune group_wait (default 30s) and group_interval (default 5m) to the operational tempo. Lower group_wait pages faster; higher group_interval reduces load.
• Set rate_limit and burst per integration. The defaults are usually too generous for noisy integrations.
• Size the AM cluster for the worst-case alert storm. A single AM instance can handle thousands of alerts per minute; a saturated dispatcher is a queue-back-pressure problem, not a CPU problem.

Production guidance

• Ship the weekly delivery canary. It catches rotated credentials before a real incident does.
• Alert on alertmanager_notifications_failed_total per integration. The reason label distinguishes config errors from transient errors and from upstream failures.
• Alert on alertmanager_cluster_health_score for any cluster of size 2 or more. A single-peer cluster has no gossip; the metric is undefined.
• Pin the AM version to the Prometheus version. AM has its own release cadence and breaking changes in the v2 API.
• Document every silence with an expiry, an owner, and a reason. The silence UI hides alerts; the comment field restores the context.

Verification

You should now be able to answer:

• What does alertmanager_notifications_total measure, and how does it differ from alertmanager_notifications_failed _total?
• What does the reason label on a failed notification tell you?
• How does alertmanager_cluster_health_score behave in a three-peer cluster when one peer becomes unreachable?
• Why is a silence that hides a critical alert a worse failure shape than a noisy alert that pages too much?
• What is the role of a synthetic delivery canary?

Quiz