Receiver Down

The alert is in AM. The route matched. The receiver was checkout-pagerduty. The notification did not land. The on-call did not get a page. The team opens AM’s nflog endpoint and sees the alert with errorState: "error" and the message notify attempt 12 failed: 401 from events.pagerduty.com. The integration key was rotated six weeks ago; the AM config still has the old key. The receiver is down. The route is fine. The rule is fine.

This lesson is the layer beyond the route. The route selected the receiver; the receiver attempted to deliver; the integration on the other end either does not answer, or answers with an error, or has been deprovisioned. AM records every attempt; the team can see the failure exhaustively once they know to look.

What “receiver down” is

A receiver-down condition in production terms is an alert that AM has accepted, routed, and tried to deliver to a configured integration, but the integration has not acknowledged the delivery. Three conditions hold:

1. The alert is in AM’s /api/v1/alerts with an active state.
2. The receiver named in the alert’s routed receiver matches the integration the team intended.
3. AM’s nflog shows the alert with errorState: "error" or with delivery attempts returning non-2xx status codes.

Condition one excludes routing defects (covered in lesson 05). Condition two excludes rule defects (lessons 03, 04). Condition three isolates the cause to the integration endpoint itself: PagerDuty Events API, Slack incoming webhook, SMTP relay, custom webhook.

A receiver-down condition is the most expensive because the symptom is invisible from the AM UI without inspecting the notification log. AM does not display the last attempt’s HTTP status on the alert detail page; it shows the alert’s state. The team must look at the nflog to see the failure.

Why a sysadmin cares

Receiver-down is the lowest-trust failure shape because it appears at the moment the team most needs the alert to arrive. The alert fires; AM attempts to deliver; the integration is unreachable; AM retries; AM eventually expires the notification. The on-call never finds out unless the team looks at AM’s log.

Three failure shapes repeat:

• Credential rotation gap. An integration key was rotated by the team responsible for the integration (the PagerDuty admin rotates a routing key; the Slack admin regenerates a webhook URL). The AM config references the old credential. The integration accepts no more requests.
• Third-party outage. PagerDuty, Slack, Opsgenie, or the email relay is having an incident of its own. AM’s retries overflow the receiver’s retry budget; the notification is dropped.
• Webhook URL change. The webhook URL was updated in the upstream system (e.g., Slack regen produced a new URL). The AM config still references the old URL; the integration returns 404.

The cost is asymmetric: a page that lands incorrectly is better than a page that does not land at all. The fix is prompt credential rotation and a synthetic test on every configuration change.

How it works

AM’s delivery machinery is a per-receiver worker pool with retry and backoff. Each notification attempt produces a sample in nflog and a log line in AM’s stderr.

+-------------------------+
| AM marks alert for      |
| delivery (group_wait    |
| elapsed)                |
+----------+--------------+
           |
           v
+-------------------------+
| Look up receiver's      |
| integration config      |
+----------+--------------+
           |
           v
+-------------------------+
| Build notification      |
| payload from template   |
+----------+--------------+
           |
           v
+-------------------------+
| HTTP POST/SMTP send     |
| to integration URL      |
+----------+--------------+
           |
           v
+-------------------------+
| Wait for response       |
+-------------------------+
   |   2xx             ! 2xx / timeout
   |                   |
   v                   v
| Mark          | Increment retry counter;
| delivered     | if (retries < max)
| in nflog;     | reschedule with
| exit          | exponential backoff
                |
                v
                +-------------------------+
                | After max retries:       |
                | mark nflog entry with    |
                | errorState: "error"      |
                +-------------------------+

The integration’s response is the gate. A successful HTTP 2xx from the integration means the receiver is up. A non-2xx, a TLS handshake failure, a DNS resolution failure, a connect timeout all count as a failed attempt. AM retries until either a 2xx arrives or the retry budget is exhausted.

The retry budget is bounded by AM’s retry_* knobs:

• group_wait is the time AM waits for additional alerts to arrive in a group before delivering.
• group_interval is the time between deliveries for the same group.
• repeat_interval is the time between repeat deliveries for an unresolved alert.
• The receiver-level retry budget is bounded by the integration’s own response.

For PagerDuty Events API v2, the integration is rate-limited to 30 publishes per integration per minute; an alert storm above that rate produces 429s. AM retries against the 429 with exponential backoff.

For Slack incoming webhooks, the URL is regenerated by Slack; the old URL returns 404. AM retries against the 404 exhaustively.

For SMTP relays, the integration is a server. AM sends; the server accepts or rejects. A 4xx from the server is a permanent failure; a 5xx is transient.

The most common cause

In roughly half of the receiver-down investigations the team reviews, the cause is an expired or rotated integration credential. The integration was healthy at last verification; a rotation event happened without updating the AM config. The integration returns 401; AM retries; the alert is dropped.

The second most common cause is a third-party outage (PagerDuty, Slack, Opsgenie). The status page indicates an incident; AM’s logs show non-2xx against the integration. The team waits for the upstream provider to recover; in the meantime, AM queues the notification.

The third most common cause is a webhook URL change (Slack specifically regenerates webhook URLs as a security measure). The new URL exists in Slack; the old URL in AM returns 404.

Under the hood

AM is a Go binary that uses an in-memory store for alerts and a separate in-memory store for the notification log (nflog). On every delivery attempt, AM records the attempt’s outcome in nflog with the HTTP status code, the integration endpoint, and the retry count. The nflog is bounded in size; older entries are evicted as new ones arrive.

The flow on a notification attempt:

1. AM resolves the receiver’s integration config.
2. AM builds the payload from the template; expands Go template variables ({{ .GroupLabels.alertname }}).
3. AM performs the HTTP call (or SMTP transaction).
4. AM records the attempt in nflog with the response code.
5. If 2xx, AM marks the alert as delivered; the entry in nflog is success.
6. If non-2xx, AM increments the retry counter; if the receiver’s retry budget is exhausted, AM marks the entry as error.

The integration’s response is the only signal that distinguishes a successful delivery from a failed delivery. AM does not have any out-of-band probe that confirms the integration is healthy between attempts. The notification is the probe.

How to configure it

An AM configuration that protects against the common receiver-down causes. Annotated example:

global:
  resolve_timeout: 5m
  # PagerDuty Events API v2 does not require SMTP globals.

route:
  receiver: default
  group_by: ['alertname', 'region']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - severity = "page"
        - team = "checkout"
      receiver: checkout-pagerduty
      continue: false

receivers:
  - name: default
    webhook_configs:
      - url: 'https://hooks.example.com/default'
        send_resolved: true
        # Send a heartbeat so the integration can detect
        # when AM is unable to reach it.
        heartbeat_interval: 5m
  - name: checkout-pagerduty
    pagerduty_configs:
      - routing_key: 'PD_CHECKOUT_KEY_REDACTED'
        send_resolved: true
        # Retry transient 5xx with exponential backoff.
        retry_on_5xx: true
    # Custom webhook for the integration status test.
    webhook_configs:
      - url: 'https://hooks.example.com/checkout-status'
        send_resolved: false

templates:
  - '/etc/alertmanager/templates/*.tmpl'

The diagnostic hooks in this config:

• routing_key: is the credential. Treat it as a secret. Rotation: update the key in PagerDuty first; then update AM; then verify with a synthetic test.
• send_resolved: true is required for pages that resolve cleanly. Without it, an alert that resolves is dropped silently.
• retry_on_5xx: true enables retry on transient upstream errors. Without it, a single 5xx is treated as a permanent failure.
• heartbeat_interval: on the default receiver is a synthetic probe. AM POSTs a heartbeat every interval; the receiver returns 2xx; the team knows AM is healthy.

The retry_on_5xx and send_resolved knobs are the two that the team most often forgets. They are not optional in production.

How to validate it

Four steps. Run all four.

Step 1: parse-time check.

amtool check-config /etc/alertmanager/alertmanager.yml

A parse error fails every receiver; the previous config continues to run.

Step 2: notification log inspection.

curl -s http://alertmanager:9093/api/v1/alerts/nflog/ \
  | jq '.entries[] | select(.groupLabels.alertname=="CheckoutHighErrorRate")'

A successful entry has status: "success". A failed entry has error and a reason like 503 from events.pagerduty.com or connection refused.

Step 3: synthetic test against the integration.

amtool alert add \
  --config.file=/etc/alertmanager/alertmanager.yml \
  --alertmanager.url=http://alertmanager:9093 \
  alertname=TestReceiver \
  severity=page \
  team=checkout \
  summary='Synthetic test alert'

A well-formed config delivers to the checkout-pagerduty receiver. A failure produces an nflog entry with the integration’s response code.

Step 4: integration status page.

For PagerDuty, check the PagerDuty status page: https://status.pagerduty.com/. For Slack, check https://status.slack.com/. For Opsgenie, the equivalent.

The status page is a coarse signal but it is the fastest when the team suspects a third-party outage.

How it can fail

Six failure shapes, each tied to a receiver-down defect:

1. Expired PagerDuty routing key. The team rotated the PagerDuty integration key without updating the AM config. Symptom: AM nflog shows notify attempt X failed: 401 from events.pagerduty.com.
2. Slack webhook URL regenerated. Slack rotated the webhook URL; the AM config still references the old URL. Symptom: AM nflog shows notify attempt X failed: 404 from hooks.slack.com.
3. PagerDuty rate limit. An alert storm produces more than 30 publishes per minute to a single integration. Symptom: AM nflog shows 429 from events.pagerduty.com; AM retries with backoff.
4. SMTP relay unreachable. The SMTP server is on a host that no longer resolves; AM’s sendmail times out. Symptom: AM log shows dial tcp: lookup smtp.example.com: no such host.
5. TLS handshake failure. The integration’s certificate expired or the CA bundle is stale on the AM host. Symptom: AM log shows x509: certificate has expired or is not yet valid.
6. Webhook returns non-2xx. A custom integration returns a 5xx on every attempt; the team did not configure retry_on_5xx. Symptom: AM nflog shows the alert as error after a single attempt.

How to troubleshoot it

Follow the six-step diagnostic.

1. Step 1, confirm AM received the alert. curl /api/v1/alerts. If the alert is not there, this lesson does not apply.
2. Step 2, confirm the route matched. amtool config routes test against a sample alert. If the route is wrong, this lesson does not apply; go to lesson 05.
3. Step 3, inspect nflog. curl /api/v1/alerts/nflog/. The error field and the reason field name the failure mode.
4. Step 4, check the integration status page. For PagerDuty, Slack, Opsgenie. If the provider has an incident, the receiver is down because the upstream is down; the fix is to wait, with the alert queued in nflog for re-delivery when the provider recovers.
5. Step 5, rotate credentials if needed. For an expired routing key, generate a new key in PagerDuty, update the AM config, restart AM, run a synthetic test.
6. Step 6, run a synthetic test. amtool alert add with a clearly-marked test alert. Confirm nflog shows success.

Security implications

Routing keys and webhook URLs are bearer credentials. A compromised routing key allows an attacker to inject alerts into the on-call’s queue; a compromised Slack URL allows arbitrary posting to the team’s Slack channel. Restrict access to the AM config file and to the integration’s administration console.

AM’s webhook sender must validate the integration’s TLS chain. A misconfigured CA bundle on the AM host silently delivers to the wrong endpoint (a TLS-stripping attacker between AM and the integration). Pin the integration’s expected certificate or use a CA bundle that AM verifies on every call.

For SMTP, the sender authenticates with the relay using credentials stored in global. A compromised AM host exposes the relay credentials. The SMTP credentials must be a service account with no other privileges.

Performance implications

AM’s delivery worker pool is single-threaded per receiver. A burst of firing alerts against a slow receiver produces a backlog in the deliver queue; the alerts sit in the group until the worker catches up. The repeat_interval and group_interval knobs are the only throttles; over- throttling produces a missed alert, under-throttling produces a backlog.

For high-volume receivers, set group_interval shorter than the integration’s average response time minus a margin. A receiver that takes 30 seconds to respond with group_interval: 30s produces back-to-back deliveries that race the integration. A group_interval: 45s gives the integration room to respond.

Production guidance

• Pin each receiver’s credentials to a file outside the AM config and reference them via environment variables. Rotation becomes a credential edit, not a config edit.
• Run amtool alert add synthetic tests after every AM config change and every credential rotation. Block changes that fail the test.
• Track the nflog error rate over time. A sudden increase is an early signal of a credential rotation gap or a third-party incident.
• Configure heartbeat_interval on the default receiver so AM’s reachability is testable independent of a firing alert.

Verification

You should now be able to answer:

• What is AM’s nflog, and where does the receiver’s last error appear?
• What is the most common cause of a receiver-down failure, and how do you distinguish it from a routing defect?
• How do you run a synthetic notification test against an AM receiver?
• Why does the integration status page matter when the nflog shows an error?

Quiz