Alertmanager Anatomy

A for: 5m rule fires in Prometheus. Two seconds later, an HTTP POST hits Alertmanager. From that moment the rule is no longer the alert — Alertmanager is. It is the state machine that decides what the alert means to a human being, who receives it, when, and how loudly. Every lesson in this module is a detail inside that one machine.

This lesson is the chassis: the loop that ingests an alert, the disk stores that survive a restart, the configuration file that shapes behaviour, and the CLI used to inspect all of it from a shell.

What it is

Alertmanager is the dispatch service in a Prometheus stack. It accepts alerts over HTTP from one or more Prometheus servers (via the Alertmanager remote protocol), removes duplicates, applies silences and inhibition, groups the survivors by label, and pushes notifications to receivers — Slack, PagerDuty, email, generic webhooks. It is a Go binary with an embedded HTTP server, an embedded web UI, and three persistent on-disk stores.

The canonical name is “Alertmanager”. The CLI shipped alongside it is amtool. The HTTP API is the “v2 API”, distinct from the deprecated v1 paths that older prometheus integrations still reach for.

Why a sysadmin cares

Three failure shapes appear when Alertmanager is treated as invisible plumbing rather than a service in its own right:

1. The “Prometheus is fine” fallacy. The alert rules fire. The alertmanagers: block points at a host that has been off for four days. The Prometheus /alerts page shows green. Nothing pages anyone. The user reports the incident instead.
2. The configuration drift trap. Two on-call engineers edit the same alertmanager.yml from a shared volume. One merges a working change; one leaves a typo. The reload happens, AM fails to parse, and the process holds the old config in memory while the new one sits on disk unvalidated.
3. The restart kills the silences panic. A config reload that requires a process restart (rather than a SIGHUP) loses the in-memory silences store. The 03:00 maintenance silence that was supposed to last until 05:00 expires at 03:04.

None of these are caught by Prometheus health probes. They are caught by understanding what Alertmanager actually is.

How it works

The dispatch loop is the model that ties every later lesson together. Read it once, then keep it in your head:

  Prometheus server(s)
       |
       |  POST /api/v1/alerts  (Alertmanager remote protocol v2)
       v
  +------------------+
  |  dispatch loop   |  <-- one goroutine per peer, ~1ms to enqueue
  +------------------+
       |
       v
  +------------------+      +------------------+
  |     nflog        |      |   silences store |
  | (active alerts)  |      |  (time-bounded   |
  |  persistent      |      |   matchers)      |
  +------------------+      +------------------+
       |                          |
       +-----------+--------------+
                   |
                   v
          +----------------+
          |  notify stage  |
          |  (group, dedup,|
          |  inhibit, route|
          |  to receiver)  |
          +----------------+
                   |
                   v
          +----------------+
          | notification   |  <-- per-receiver, timeouts, retries
          | log            |
          +----------------+
                   |
                   v
          Slack / PagerDuty / webhook / email

The salient property is that the active alerts (nflog), silences, and notification log are independent stores. A SIGHUP reload does not touch them. A restart does not touch the on-disk versions, but it does drop the in-memory caches of the same stores until AM replays them.

Under the hood

Alertmanager 0.28.x is a single static Go binary. The behaviour is shaped by a YAML file passed at start (--config.file=). The three persistent stores are directories under --storage.path (default /data):

/data
├── nflog/                    # active firing alerts, last 120 minutes by default
│   ├── 00012345
│   └── 00012346
├── silences/                 # all silences, active and expired
│   ├── 00000001
│   └── 00000002
└── notify/                   # per-receiver notification log
    └── <receiver-hash>/
        ├── 00000001
        └── 00000002

• nflog/ records every alert that has been firing in the recent window. It is the structure that powers “how long has this been firing?” and the for: evaluation inside routes (repeat_interval). The window is bounded by --data.retention=120h by default for state; nflog itself uses a much shorter window (default 5h since 0.28).
• silences/ is the on-disk mirror of the in-memory silence index. Silences survive a restart because every silence is written here as it is created or modified.
• notify/ is a per-receiver append-only log of notifications attempted, with the resolved state of the group at the time of the attempt. It is the structure the UI uses to show “what was sent and when”.

Process model: one binary, one event loop. Routes are evaluated in-process. Cluster mode is achieved with a gossip protocol on a dedicated port (--cluster.listen-address=) so multiple AM nodes share state without a database.

How to configure it

The minimum viable alertmanager.yml for a single-node install:

global:
  resolve_timeout: 5m
  smtp_smarthost: 'smtp.internal.example.com:587'
  smtp_from: 'alertmanager@example.com'
  smtp_auth_username: 'alertmanager@example.com'
  smtp_auth_password: 'file:///etc/alertmanager/smtp.pass'

route:
  receiver: 'default-slack'
  group_by: ['alertname', 'cluster']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes: []

receivers:
  - name: 'default-slack'
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/T0000/B0000/XXXX'
        channel: '#oncall-alerts'
        send_resolved: true

inhibit_rules: []

templates:
  - '/etc/alertmanager/templates/*.tmpl'

Notes on the keys that matter operationally:

• resolve_timeout (default 5m) is how long an alert can be resolved before AM stops treating the resolution as worth notifying. Tighten it for low-noise; loosen it when transient flaps are common.
• group_wait, group_interval, and repeat_interval are the three timers of the notify loop. They are covered in detail in the grouping lesson.
• templates: loads Go template files from disk at startup. Missing templates are logged but do not crash the process.
• api_url for Slack and the PagerDuty routing key are secrets. They belong in a separate file mounted with restricted permissions, or in a secrets manager.

How to validate it

Three independent checks, run in order. The first two are read-only. The third is configuration-impact (it writes to AM memory; it does not drop alerts).

# 1. Validate the YAML schema without contacting Alertmanager.
#    Severity: READ-ONLY.
amtool check-config /etc/alertmanager/alertmanager.yml
# Output (success):
# Checking 'alertmanager.yml'...
# SUCCESS

# 2. Show the parsed configuration back to you.
#    Severity: READ-ONLY.
amtool --alertmanager.url=http://localhost:9093 config show
# Output (truncated):
# route:
#   receiver: default-slack
#   group_by:
#   - alertname
#   - cluster
# ...

# 3. Smoke-test a synthetic alert against the running instance.
#    Severity: CONFIGURATION (writes an alert to AM memory).
amtool --alertmanager.url=http://localhost:9093 alert add \
  alertname=TestAlert severity=warning cluster=staging
# Output:
# level=info ts=2026-08-14T02:00:00Z caller=coordinator.go:... 
#   component=active stage=active alerts=[TestAlert]

The HTTP API equivalents, useful for scripting:

# List currently firing alerts.
# Severity: READ-ONLY.
curl -s http://localhost:9093/api/v2/alerts | jq '.[].labels.alertname'
# Output:
# "HighRequestLatency"
# "DiskFillingSoon"

# Confirm the service is up. Used by load balancers and probes.
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:9093/-/healthy
# Output: 200

# Confirm the cluster (if any) is converged.
curl -s http://localhost:9093/api/v2/status | jq '.cluster.status'
# Output: "ready"

To reload the configuration without dropping in-flight notifications, send SIGHUP. The process reloads routes, receivers, templates, and inhibit rules atomically. The in-memory state of nflog, silences, and the notification log is preserved:

# Reload. Severity: SERVICE-IMPACT (briefly interrupts evaluation).
kill -HUP $(pidof alertmanager)

How it can fail

The recurring failure modes, in descending order of operational cost:

1. Alertmanager host unreachable from Prometheus. Prometheus still fires rules. /alerts shows them as firing. Nothing reaches the on-call channel. Detect with prometheus_target_status{job="alertmanager"} — keep the Prometheus scrape config for Alertmanager itself, and alert on up == 0.
2. Bad config on reload. A typo in routes: causes amtool check-config to exit non-zero. A SIGHUP would still succeed if the new YAML parses, but the route tree is now wrong. Always run amtool check-config before SIGHUP, and keep the previous file as alertmanager.yml.bak so a rollback is one move.
3. Storage path full. nflog, silences, and notify are append-only directories. With a small disk, notify/<hash>/ for a chatty receiver can grow into gigabytes. AM will not refuse to write, but the underlying fsync will start returning errors. Watch disk usage on the storage path explicitly.
4. Cluster gossip partition. Two AM nodes that cannot reach each other do not split-brain the alert stream — they each assume the other is down and notify. The result is duplicate pages during a network blip. This is by design (better duplicate than miss) but it has a real cost.
5. api_url rotated in the upstream but not in the YAML. Slack and PagerDuty surface this as a 404 in the AM log and a growing notification log entry per firing alert. It is a slow leak; check the AM logs at every receiver rotation.
6. Version drift between prometheus and alertmanager. The Alertmanager remote protocol has had breaking changes. Pin compatible versions in your platform manifest and verify the handshake with promtool and the AM log line that confirms peer negotiation.

How to troubleshoot it

When the on-call channel is silent, the order matters. Diagnose before changing state.

1. Is Alertmanager up? curl -s http://am:9093/-/ready. If not, look at the process and the most recent stderr. The /metrics endpoint is also a good probe — if it returns 200, the binary is alive even if the UI is not.
2. Is Prometheus reaching it? On the Prometheus host, look at the alertmanagers/ discovery target and the prometheus_target_status metric. A value of 0 for the Alertmanager job means Prometheus itself cannot connect.
3. Is AM receiving alerts? curl -s http://am:9093/api/v2/alerts. If the list is empty but Prometheus shows firing alerts, the problem is on the Prometheus side (wrong alertmanagers: endpoint, wrong scheme, expired TLS).
4. Are the alerts being routed? amtool config routes test is the workhorse here — see the routing tree lesson for the exact invocation.
5. Is the receiver reachable from AM? Tail the AM log while sending a synthetic alert. A dial tcp: timeout means the problem is downstream of AM entirely.

Security implications

Alertmanager exposes four surfaces:

• HTTP API (/api/v2/*) — write-capable on the alert, silences, and (in older versions) the config endpoints. Bind it to the cluster network, never to a public address. The v2 API is unauthenticated by default; front it with a reverse proxy that does TLS termination and basic auth if you expose the UI.
• Web UI — same surface as the API. Same controls apply.
• Cluster gossip — --cluster.listen-address and --cluster.peer. Encrypt with --cluster.tls-* flags if the network is not trusted. The gossip traffic is unencrypted by default.
• Webhook receivers — the URL is fetched from the YAML. Do not put a credential inside api_url that grants write access to production systems; a single misrouted webhook becomes an attack vector into whatever it points at.

Secrets belong in a file mounted with 0600 permissions and read via the file:// scheme, or in a secrets manager fetched at start. The YAML itself is checked into source control and read by everyone with read access to the repo.

Performance implications

The hot paths are the notify loop (one goroutine per receiver, re-evaluating every group_interval) and the API write path (POST /api/v2/silences, POST /api/v1/alerts). Two failure shapes appear at scale:

• Receiver-side rate limiting. Slack, PagerDuty, and email providers will rate-limit or temporarily reject noisy Alertmanagers. The right discipline is group_by on labels that have low cardinality in normal operation (alertname, cluster, severity), and a group_interval that is longer than the typical incident duration for that severity.
• High-cardinality nflog. A label that takes thousands of unique values per firing alert (request IDs, container IDs) will explode the nflog size and the notification log per group. The label hygiene discipline lives in the rules, not in AM — but AM is where it becomes painful.

Capacity planning: a single AM node on modest hardware (1 vCPU, 1 GiB RAM) handles tens of thousands of firing alerts in nflog without trouble. The disk stores are the limit, not the RAM.

Production guidance

• Pin a version in your platform manifest. Alertmanager 0.27.x changed the default retention of nflog. 0.28.x has further API refinements. Document the version you run and read the upgrade notes before bumping.
• Always run amtool check-config before SIGHUP. Make it part of the deploy pipeline, not an after-thought.
• Probe /metrics from Prometheus on a 15-second interval. Alert on up == 0. Probe /api/v2/status and alert on cluster.status != "ready" if you run a cluster.
• Back up the storage path. Silences and the notification log are operational state. They survive a process restart, but not a rm -rf /data (intentional or otherwise).
• Document the runbook for “AM is down”. Include the SIGHUP command, the path to the previous alertmanager.yml, and the Slack channel to post in when paging is paused.

Verification

You should now be able to answer:

• What three on-disk stores does Alertmanager maintain, and what does each one persist?
• How do you reload the configuration without dropping in-flight notifications?
• What is the difference between amtool check-config and amtool config show?
• What does GET /api/v2/status tell you that GET /-/healthy does not?
• Why does a cluster partition cause duplicate pages rather than dropped pages?

Quiz