Alerting Section

An alert fires in the Prometheus UI. The rule is healthy, the state is firing, the expression is unambiguous. Nobody is paged. The post-incident review finds a one-line cause: the alerting: block still pointed at an Alertmanager DNS name retired three months earlier, and nobody was watching prometheus_notifications_errors_total. The rules were perfect. The delivery path was not. Most “alerting is broken” incidents are exactly this — not bad expressions, but a bad or unwatched path between Prometheus and Alertmanager.

The alerting: block is that path. It is small, it is configured once, and it is the difference between an alert that exists and an alert that reaches a human.

What the alerting block is

A precise division of labour:

• Prometheus evaluates rules, tracks pending and firing state, and hands firing alerts to its notify layer.
• Alertmanager groups, deduplicates, silences, routes and actually notifies (e-mail, Slack, PagerDuty, webhooks).

The alerting: block tells the notify layer where the Alertmanagers are and what may leave the building:

alerting:
  alert_relabel_configs:
    # Drop informational alerts before they ever reach Alertmanager.
    # Every drop rule needs a comment and an owner.
    - source_labels: [severity]
      regex: 'info'
      action: drop

  alertmanagers:
    - scheme: https
      timeout: 10s          # default; raise only for slow links
      api_version: v2       # the default in 2.55; v1 is removed in 3.x
      static_configs:
        - targets:
            - alertmanager-1.example.com:9093
            - alertmanager-2.example.com:9093
      tls_config:
        ca_file: /etc/prometheus/tls/ca.crt
      # basic_auth / authorization / oauth2 are available here too.
      # Alternatives to static_configs:
      #   dns_sd_configs   - a headless service or SRV record
      #   file_sd_configs  - file-driven inventories
      #   kubernetes_sd_configs, ec2_sd_configs, ...
      # relabel_configs ON THIS ENTRY selects which discovered
      # Alertmanagers to talk to. It filters destinations, not alerts.

Three semantics that matter more than any field:

1. Prometheus fans out to every discovered Alertmanager. There is no load balancing. Two Alertmanagers means each firing alert is POSTed to both, and Alertmanager’s clustering handles the rest.
2. alert_relabel_configs rewrites alert content; the entry’s relabel_configs rewrites destination selection. Confusing the two is a classic: a label filter meant for alerts placed under the Alertmanager entry silently selects no Alertmanagers at all.
3. External labels attach here. Outbound alerts pick up the server’s external_labels where the alert does not already carry those names — this is what makes an HA pair’s alerts identical and deduplicable.

alert_relabel_configs: the last checkpoint before the wire

Applied per alert, before external labels fill gaps and before the queue. Uses that earn their keep:

• Dropping a class at source. severity: info alerts that exist for dashboards but must never page.
• Stripping sensitive labels. A label that carries a user ID or a token fragment is removed here so it never leaves the perimeter in a notification payload.
• Normalising identity. Rewriting a legacy env label into the environment your Alertmanager routes expect.

The corresponding hazard: a drop rule that matches more than intended (regex: '.*' after a hasty edit) makes alerts vanish by design — no error, no retry, no metric that says “you deleted your paging path”. The alert shows firing in Prometheus and Alertmanager never sees it. Every drop rule gets a comment, an owner, and a test.

Validating the delivery path

# 1. Which Alertmanagers does the running server see?
curl -s http://localhost:9090/api/v1/alertmanagers | jq .data
# {
#   "activeAlertmanagers": [
#     { "url": "https://alertmanager-1.example.com:9093/api/v2/alerts" },
#     { "url": "https://alertmanager-2.example.com:9093/api/v2/alerts" }
#   ],
#   "droppedAlertmanagers": []
# }

# 2. Is anything actually firing right now?
curl -s http://localhost:9090/api/v1/alerts | \
  jq -r '.data.alerts[] | [.labels.alertname, .state] | @tsv'

# 3. Is the notify layer healthy? (PromQL on the server itself)
#    rate(prometheus_notifications_errors_total[5m])
#    prometheus_notifications_dropped_total
#    prometheus_notifications_queue_length
#      / prometheus_notifications_queue_capacity

The end-to-end proof is a synthetic alert: a trivially true expression in a staging-tagged rule (expr: vector(1) with a severity: heartbeat label), watched from Prometheus state through Alertmanager to a test receiver. Run it permanently. The day it stops arriving is the day your delivery path broke — found by monitoring, not by a real incident going unnoticed.

How the alerting section fails

1. Dead destination. Retired DNS name, renumbered IP, wrong port. Symptom: prometheus_notifications_errors_total climbing, log lines Error sending alert, alerts firing in the UI, silence on every receiver.
2. Alertmanager target selected away. A relabel_configs on the Alertmanager entry drops all discovered targets. Symptom: activeAlertmanagers is empty in the API; notifications go nowhere and nothing errors, because there is nothing to error against.
3. Over-broad alert_relabel_configs drop. Symptom: alerts fire in Prometheus, never appear in Alertmanager, and all notification metrics look calm. The most silent failure this block can produce.
4. HA pair with different external labels. Symptom: every incident pages twice; the “duplicates” differ by one label. Alertmanager deduplication requires identical label sets — this is the global settings lesson arriving with interest.
5. Timeout too tight for the link. A remote Alertmanager behind a congested path, timeout: 10s left as default. Symptom: intermittent context deadline exceeded, alerts arriving late or in bursts after retries.
6. Scheme or TLS mismatch. scheme: https against an HTTP-only Alertmanager, or a CA the server does not trust. Symptom: handshake or HTTP response to HTTPS client errors in the log, errors metric rising, zero deliveries.

Troubleshooting, in order

1. Is the alert firing? /api/v1/alerts. If it is not there, the problem is in rules, not delivery.
2. Does Prometheus know any Alertmanagers? /api/v1/alertmanagers. Empty means discovery or relabeling failure on the entry.
3. Are sends failing? prometheus_notifications_errors_total rate, plus the log. The error text names the cause: refused, timeout, TLS, 4xx from the far side.
4. Are alerts being dropped before the queue? Diff the firing set against what Alertmanager shows (amtool or its API). Alerts present in Prometheus but never in Alertmanager, with healthy send metrics, means alert_relabel_configs ate them.
5. Is the far end Acknowledging but not paging? Then the failure has moved into Alertmanager routing or receivers — a different lesson, and a good outcome for this one.

Security implications

Alert payloads are operational intelligence: hostnames, label values, annotation text with $labels interpolated, sometimes query fragments. They cross the network to Alertmanager — send them over TLS with verification on, and authenticate where the receiver supports it (basic_auth, authorization, oauth2 on the entry). Use alert_relabel_configs to strip labels that carry user identifiers or anything resembling a secret before payloads leave your network. And scope who can change this block: redirecting alertmanagers targets is redirecting your paging to an attacker’s collector.

Performance implications

Quiet alerting costs almost nothing. Storms are the test: thousands of simultaneously firing alerts fill the 10,000-entry notify queue, and overflow shows as prometheus_notifications_dropped_total — dropped means never sent, even though the alert kept firing. The mitigations are upstream (better for: hygiene, aggregation alerts instead of per-target fan-out) and at the relabel checkpoint (drop classes that must not page). Watch queue length against capacity during any major incident; a full queue is how big outages eat their own pages.

Production guidance

• Run at least two Alertmanagers and configure both statically, even when they cluster. Fan-out is your redundancy; clustering is theirs.
• Set api_version: v2 explicitly so the 3.x migration is a non-event.
• Alert on prometheus_notifications_errors_total, prometheus_notifications_dropped_total, and an empty activeAlertmanagers set. The delivery path is production-critical infrastructure.
• Every alert_relabel_configs drop carries a comment, an owner, and a test. Prefer dropping in Alertmanager (where it is visible) unless the volume or the sensitivity justifies source drops.
• Keep a synthetic heartbeat alert firing forever through the real path, and page if it stops arriving.

Verification

You should now be able to answer:

• Which component actually pages a human, and what exactly does Prometheus POST when an alert fires?
• What is the difference between alert_relabel_configs and the relabel_configs on an Alertmanager entry?
• Why does an HA pair need identical external labels for deduplication to work?
• Which three metrics tell you the delivery path is broken before a user does?
• What happens to alerts that fire and resolve during a full Alertmanager partition?

Quiz