Alert receivers — Slack, PagerDuty, and webhook

The Alertmanager receivers are the destinations for the alerts. Slack, PagerDuty, email, and webhook are the canonical receivers. The templates are the Go templates for the messages. This lesson walks the receivers, the templates, and the production patterns.

The Slack receiver

The Slack receiver:

receivers:
- name: slack-warnings
  slackConfigs:
  - apiURL: https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX
    channel: "#alerts-warnings"
    title: "{{ .CommonAnnotations.summary }}"
    text: "{{ .CommonAnnotations.description }}"
    sendResolved: true

The Slack receiver sends the alerts to the Slack channel.

The PagerDuty receiver

The PagerDuty receiver:

receivers:
- name: pagerduty-critical
  pagerdutyConfigs:
  - serviceKey: <integration-key>
    severity: critical
    description: "{{ .CommonAnnotations.summary }}"
    details:
      - firing: "{{ .Alerts.Firing | len }}"
      - resolved: "{{ .Alerts.Resolved | len }}"
      - cluster: "{{ .CommonLabels.cluster }}"

The PagerDuty receiver sends the alerts to the on-call.

The email receiver

The email receiver:

receivers:
- name: email-digest
  emailConfigs:
  - to: ops@example.com
    from: alertmanager@example.com
    smarthost: smtp.example.com:587
    authUsername: alertmanager@example.com
    authPassword: <password>
    headers:
      Subject: "{{ .CommonAnnotations.summary }}"
    sendResolved: true

The email receiver sends the alerts as a digest.

The webhook receiver

The webhook receiver:

receivers:
- name: webhook-custom
  webhookConfigs:
  - url: http://my-service:8080/alerts
    sendResolved: true

The webhook receiver sends the alerts to a custom URL.

The templates

The templates:

templates:
- /etc/alertmanager/templates/*.tmpl

receivers:
- name: slack-warnings
  slackConfigs:
  - apiURL: ...
    channel: ...
    title: '{{ template "slack.default.title" . }}'
    text: '{{ template "slack.default.text" . }}'

The templates are the Go templates for the messages.

The template example

The template example:

{{ define "slack.default.title" }}
[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonLabels.alertname }}
{{ end }}

{{ define "slack.default.text" }}
{{ range .Alerts }}
*Alert:* {{ .Annotations.summary }}
*Description:* {{ .Annotations.description }}
*Severity:* {{ .Labels.severity }}
*Runbook:* {{ .Annotations.runbook_url }}
{{ end }}
{{ end }}

The template is the message format.

The AlertmanagerConfig per receiver

The AlertmanagerConfig per receiver:

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: my-config
spec:
  route:
    receiver: default
    routes:
    - matchers:
      - severity = critical
      receiver: pagerduty-critical
  receivers:
  - name: pagerduty-critical
    pagerdutyConfigs:
    - serviceKey: <key>
      severity: critical

The AlertmanagerConfig is the configuration.

The secrets for the receivers

The secrets for the receivers:

apiVersion: v1
kind: Secret
metadata:
  name: alertmanager-receiver-secrets
stringData:
  slack-webhook-url: https://hooks.slack.com/services/...
  pagerduty-service-key: <key>

The secrets are the credentials.

The production patterns

The production patterns:

flowchart LR
    A[Alert] --> B{severity?}
    B -->|critical| C[PagerDuty on-call]
    B -->|warning| D[Slack channel]
    B -->|info| E[Email digest]
    C --> F[Page]
    D --> G[Thread]
    E --> H[Daily digest]

The pattern is the production routing.

The cross-course references

• The Alertmanager course covers the routing.
• The Slack / PagerDuty courses cover the receivers.
• The Prometheus course covers the alerting.

Quiz

Production discipline

• Use the canonical receivers. Slack, PagerDuty, email.
• Use the templates. The messages.
• Send the resolved notifications. The operator’s confirmation.
• Use the secrets. The credentials.
• Document the receivers. The destinations, the templates.
• Test the receivers. Send a test alert.

The receivers are the alert destinations. Operating it well is the canonical receivers, the templates, and the production patterns.