Receivers and Templates

A critical alert routes to the pagerduty-oncall receiver. The PagerDuty integration key is correct. The Slack webhook is correct. Both are stored in a secrets/ directory on the Alertmanager host with 0600 permissions. The template renders the alert name, severity, cluster, instance, a one-line summary, and a link to the runbook. The receiver sends a Slack message and a PagerDuty incident within ten seconds of the alert firing.

This is what a working receiver looks like. Every receiver is the boundary between the alerting pipeline and an external service that you do not control. Get the boundary wrong and the right page never arrives, or arrives in the wrong shape, or arrives and times out.

What it is

A receiver: is the named destination in alertmanager.yml that a route points at. Each receiver contains one or more *_configs blocks, one per integration type. The supported integrations in Alertmanager 0.28.x are:

• slack_configs — Slack incoming webhooks (channel override, thread title, mentions).
• pagerduty_configs — PagerDuty Events API v2 (routing key, severity, source, dedup_key).
• opsgenie_configs — Opsgenie alerts API.
• webhook_configs — generic HTTP POST with a JSON body. The most flexible and the most failure-prone.
• email_configs — SMTP with optional auth and TLS.
• pushover_configs, victorops_configs, wechat_configs, sns_configs — niche integrations.
• msteams_configs — community-maintained via the generic webhook.

Templates are separate. They are Go templates loaded from disk at startup and referenced by name from each *_configs block. The default templates render a reasonable Slack message and a reasonable PagerDuty incident; production-grade alerts usually override them with team-specific templates that include a runbook link and an owner.

Why a sysadmin cares

Receivers are the boundary between the alerting pipeline and external systems. The boundary has three classes of failure:

1. The wrong key. A Slack incoming webhook was rotated in the Slack workspace admin console but not in alertmanager.yml. Notifications return 404. The notification log fills with errors. The on-call channel is silent.
2. The too-long timeout. A webhook receiver points at a service that accepts the TCP connection and then hangs. Alertmanager waits the default 10 seconds per webhook before timing out. A burst of 20 alerts with 5 webhooks each is 100 ten-second waits; the notification pipeline stalls.
3. The template panic. A Go template references a label that does not exist on every alert. AM logs a panic and skips the notification. The on-call never receives the alert.

The discipline is to test every receiver end-to-end with a synthetic alert, to keep secrets out of the YAML, and to set a tight timeout: on every webhook receiver.

How it works

  group G (decided by the routing tree)
    |
    v
  +----------------------+
  |   templates          |  <-- Go template files loaded at startup
  |   - default.tmpl     |
  |   - slack.tmpl       |
  |   - pagerduty.tmpl   |
  +----------------------+
    |
    v
  +----------------------+
  |   receiver senders   |  <-- one goroutine per integration
  |   - slack_configs    |
  |   - pagerduty_configs|
  |   - webhook_configs  |
  +----------------------+
    |
    v
  External services (Slack, PagerDuty, custom HTTP)

Each receiver has its own goroutine pool (bounded by --webhook.timeout and the receiver-level timeout:). Notifications for a single group are sent in parallel; multiple groups for the same receiver are serialised through the receiver’s notification queue.

Under the hood

A webhook_configs block POSTs a JSON body to the configured URL. The default body includes the alert labels, annotations, status, and a top-level structure suitable for human reading. Custom templates can override the body entirely.

A pagerduty_configs block posts to https://events.pagerduty.com/v2/enqueue with a routing key, a dedup_key (defaults to alertname + fingerprint), and the alert summary. The routing key identifies the PagerDuty service to page; the dedup key identifies the incident inside the service. Two alerts with the same dedup_key de-duplicate into the same PagerDuty incident.

A slack_configs block POSTs a JSON body to the configured incoming webhook URL. The default body is a simple text message; custom templates can render Slack blocks with buttons and fields. The channel: override lets a single Slack webhook target multiple channels.

A send_resolved: flag controls whether AM sends a “resolved” notification when an alert transitions from firing to resolved. The default is false. For paging receivers (PagerDuty), set send_resolved: true so the on-call sees the resolution.

A timeout: on each *_configs block sets the per-request timeout. Default 10 seconds. Five is more honest for upstream services that hang.

A retry on a 5xx response is automatic for some integrations. For webhook_configs, AM retries up to the --webhook.max-concurrent limit; a 4xx is not retried (it is a permanent failure for that body).

How to configure it

A realistic receivers: block for a multi-team fleet:

global:
  resolve_timeout: 5m

templates:
  - '/etc/alertmanager/templates/*.tmpl'

receivers:
  # Slack: low-friction, good for warnings and info.
  - name: 'slack-default'
    slack_configs:
      - api_url_file: '/etc/alertmanager/secrets/slack-default.url'
        channel: '#oncall-default'
        send_resolved: true
        timeout: 5s
        title: '{{ .CommonLabels.alertname }} - {{ .CommonLabels.severity }}'
        text: |
          {{ range .Alerts }}
          *{{ .Labels.alertname }}* on `{{ .Labels.instance }}` ({{ .Labels.cluster }})
          {{ .Annotations.summary }}
          Runbook: https://runbooks.example.com/{{ .Labels.alertname }}
          {{ end }}

  # PagerDuty: paging, critical only.
  - name: 'pagerduty-oncall'
    pagerduty_configs:
      - routing_key_file: '/etc/alertmanager/secrets/pd-oncall.key'
        send_resolved: true
        severity: 'critical'
        source: 'alertmanager.prod-eu-1'
        description: |
          {{ .CommonLabels.alertname }} on {{ .CommonLabels.cluster }}
          {{ .CommonAnnotations.summary }}
        details:
          cluster: '{{ .CommonLabels.cluster }}'
          runbook_url: 'https://runbooks.example.com/{{ .CommonLabels.alertname }}'
        timeout: 5s

  # Webhook: integration with a custom incident-response tool.
  - name: 'webhook-incident-bridge'
    webhook_configs:
      - url: 'https://incidents.internal.example.com/ingest'
        send_resolved: true
        max_alerts: 50
        timeout: 5s
        # The default body is replaced by the named template.
        # If not specified, AM renders a JSON body with the
        # alert labels, annotations, and status.

  # Email: audit trail for low-severity and compliance.
  - name: 'email-audit'
    email_configs:
      - to: 'sre-audit@example.com'
        from: 'alertmanager@example.com'
        smarthost: 'smtp.internal.example.com:587'
        auth_username: 'alertmanager@example.com'
        auth_password_file: '/etc/alertmanager/secrets/smtp.pass'
        require_tls: true
        send_resolved: true
        headers:
          Subject: '[ALERT] {{ .CommonLabels.alertname }} - {{ .CommonLabels.severity }}'
        html: |
          <h2>{{ .CommonLabels.alertname }}</h2>
          <p><b>Cluster:</b> {{ .CommonLabels.cluster }}</p>
          <p><b>Summary:</b> {{ .CommonAnnotations.summary }}</p>
          <pre>{{ range .Alerts }}{{ .Labels.YAML }}
          {{ end }}</pre>

Three things to read off this:

• Secrets via _file suffix. Every integration uses api_url_file, routing_key_file, or auth_password_file rather than embedding the credential in the YAML. The YAML is safe to commit; the secrets directory is 0700 and lives on a different volume or in a secrets manager.
• Tight timeout: 5s. The default 10s is too long for upstream services that hang. Five is more honest.
• send_resolved: true on paging receivers. The on-call needs to see the resolution, not just the page.

A minimal template file at /etc/alertmanager/templates/slack.tmpl:

{{ define "slack.default.text" }}
{{ range .Alerts }}
*{{ .Labels.alertname }}* on `{{ .Labels.instance }}`
Cluster: {{ .Labels.cluster }}
Severity: {{ .Labels.severity }}
Summary: {{ .Annotations.summary }}
Runbook: https://runbooks.example.com/{{ .Labels.alertname }}
{{ end }}
{{ end }}

The define block names the template; the receiver config references it by name.

How to validate it

amtool check-config validates the YAML. To validate the receiver end-to-end, fire a synthetic alert and watch the notification log:

# 1. Fire a synthetic alert.
amtool --alertmanager.url=http://localhost:9093 alert add \
  alertname=TestAlert severity=critical cluster=prod-eu-1 instance=db-1
# Output:
# level=info ts=... caller=coordinator.go:... component=active
#   stage=active alerts=[TestAlert]

# 2. Tail the AM log.
journalctl -u alertmanager -f
# Expected lines (truncated):
# level=info ts=... caller=notify.go:... receiver=slack-default
#   group=TestAlert/...
# level=info ts=... caller=notify.go:... receiver=pagerduty-oncall
#   group=TestAlert/...

# 3. Inspect the notification log on disk.
ls -la /data/notify/slack-default/
# Output:
# -rw-r--r-- 1 alertmanager alertmanager 4321 Aug 14 02:00 00000001

# 4. Confirm the notification reached Slack (or the upstream).
#    Slack: open the channel, look for the message.
#    PagerDuty: check the on-call schedule for the incident.

# 5. Resolve the synthetic alert.
amtool --alertmanager.url=http://localhost:9093 alert add \
  alertname=TestAlert severity=critical cluster=prod-eu-1 instance=db-1 \
  --end
# Expected: AM sends a "resolved" notification if send_resolved: true.

For the template itself, amtool config show does not render. The right approach is to fire a synthetic alert and inspect the rendered message at the receiver end.

For the API view:

# List all receivers and their integration counts.
curl -s http://localhost:9093/api/v2/receivers | jq '.[].name'
# Output:
# "slack-default"
# "pagerduty-oncall"
# "webhook-incident-bridge"
# "email-audit"

How it can fail

The recurring failure modes, in descending order of operational cost:

1. Rotated credential. A Slack incoming webhook was rotated in Slack but not in alertmanager.yml. Every notification returns 404. The notification log fills; the channel is silent. The fix is to rotate the credential everywhere and to monitor the 4xx rate via the AM log.
2. Template panic. A Go template references .Labels.runbook_url which is not present on every alert. AM logs a template panic and skips the notification. The fix is to use {{ .Labels.runbook_url | default "https://runbooks.example.com/" }} or to remove the reference.
3. Webhook hang. A custom webhook receiver points at a service that accepts the TCP connection and then hangs. Every notification waits the timeout before giving up. The fix is to tighten timeout: 5s and to alert on AM-side latency metrics.
4. Missing send_resolved: true. The on-call sees the page but never sees the resolution. The incident lingers in PagerDuty; the SLO is met on AM’s side but not on the human side. Set send_resolved: true on every paging receiver.
5. PagerDuty routing key mismatch. A PagerDuty integration was created for a different service; the routing key in the YAML points at the wrong service. Pages arrive in the wrong rotation. Validate against the PagerDuty service directory before saving.
6. Email SMTP TLS failure. The SMTP server requires TLS but the receiver config does not set require_tls: true. The email is silently dropped; the notification log shows a TLS handshake error. Set require_tls: true and verify with openssl s_client from the AM host.

How to troubleshoot it

When a notification does not arrive, the diagnostic order:

1. Is the receiver defined? amtool config show | grep <receiver-name>. If not present, the routing tree points at a non-existent receiver.
2. Is the alert being routed to the receiver? Inspect GET /api/v2/alerts for the alert and check .receivers.
3. Is the notification log entry being written? ls -la /data/notify/<receiver-hash>/. An entry that is growing confirms AM is trying to send; a missing entry confirms the alert is not being routed to this receiver.
4. Is the upstream returning an error? Tail the AM log while a synthetic alert fires. A line containing caller=notify.go:... err=... shows the upstream response.
5. Is the template rendering correctly? Add a debug webhook_configs block pointing at a request bin (e.g. webhook.site) and fire a synthetic alert. Inspect the rendered body.

Security implications

Receivers carry the security-sensitive surface of Alertmanager. The controls:

• Secrets via file references. Use api_url_file, routing_key_file, auth_password_file — never embed credentials in the YAML.
• Secrets directory permissions. 0700 for the directory, 0600 for each file, owned by the alertmanager user. The YAML is in source control; the secrets are not.
• Webhook URL allowlist. A webhook_configs URL that points at an attacker-controlled service is a write vector. Validate every webhook URL against an allowlist.
• Email SMTP credentials. SMTP passwords are equivalent to the SMTP user’s privileges. Treat them as such.
• PagerDuty routing keys. A routing key grants the right to page a service. Compromise of the key is a paging abuse vector.

The right discipline is to keep the YAML in source control with no secrets, the secrets in a separate volume or secrets manager, and the webhook URLs in an allowlist.

Performance implications

Receivers are the bottleneck of the alerting pipeline. The hot path is the upstream HTTP request. The constraints:

• Slack incoming webhooks: rate-limited per workspace. A burst of 100 alerts in a group_interval window hits the rate limit and receives 429s. The fix is grouping granularity.
• PagerDuty Events API v2: rate-limited per integration. Same discipline.
• Custom webhooks: depend on the upstream. A 30-second timeout on a hung upstream is 30 seconds of goroutine blocked.
• Email SMTP: depends on the MTA. A failed SMTP server backs up the email receiver’s notification queue.

The receiver-level timeout: is the right lever. The group_interval and group_by (covered in the grouping lesson) are the upstream discipline.

Production guidance

• Use file references for every credential. api_url_file, routing_key_file, auth_password_file. Never embed.
• Tight timeout: 5s on every receiver. The default 10s is too long for upstream services that hang.
• Set send_resolved: true on every paging receiver. The on-call needs to see the resolution.
• Validate end-to-end after every change. Fire a synthetic alert with amtool alert add and confirm at the receiver end.
• Audit receivers quarterly. Enumerate the receivers in alertmanager.yml, confirm each still points at a working upstream, and rotate credentials on a schedule.
• Document the runbook for “receiver is silent”. Include the AM log line to look for, the secret file to check, and the synthetic alert command to fire.

Verification

You should now be able to answer:

• What is the difference between an integration URL and a routing key?
• Why should every credential use a _file reference rather than being embedded in the YAML?
• What does send_resolved: true do, and why should every paging receiver set it?
• What is the right timeout: for a webhook_configs block, and why?
• How do you validate a receiver end-to-end without firing a real alert?

Quiz