Alertmanager Backup

The on-call engineer was paged at 02:14. A database had failed over to a standby in another region. The team had silenced the standby’s health alerts four hours earlier, expecting the failover to take 90 minutes. The Alertmanager restarted at 02:11 on a routine config reload that triggered a peer-state mismatch. The silences were in memory only. The pages started firing again at 02:14 for an outage the team was already handling.

This lesson is the Alertmanager backup. The right shape is two artefacts: a configuration file in Git, and a periodic export of the active silences to durable storage. The configuration is the primary source for routing and receivers; the silence export is the safety net for in-flight acknowledgements.

What it is

An Alertmanager backup is a copy of the durable state that Alertmanager needs to serve the same alerts and silences from a recovered cluster. The three states in Alertmanager:

• Configuration file. Routing tree, receivers, inhibit rules, templates. Lives in Git by default. The Git history is the backup.
• Silences. In-memory state that suppresses alerts matching a set of matchers. Created via the API or amtool silence. Lost on restart unless persisted.
• Notification log (nflog). In-memory history of notifications sent. Used for dedup and grouping across restarts. Default retention is short; the storage path can be set but the default is in-memory.

The discipline is to ship the configuration file from Git, to periodically export the active silences to durable storage, and to verify the recovery path with a quarterly drill.

Why a sysadmin cares

Alertmanager is the page-the-human layer. A failure here means:

• Pages either stop firing (an alert that no one wakes up to) or fire in storms (every alert that was being silenced comes back at once).
• The deduplication state is reset. Two duplicate alerts become four.
• The grouping state is reset. A multi-target outage produces a flood instead of one grouped page.

A working backup turns a cluster failure into a 15-minute restoration. A broken backup turns the same failure into a paging storm and a confused on-call engineer.

How it works

   Prometheus (or Loki ruler, or Grafana unified alerting)
        |
        v
   Alertmanager
        |
        +-- configuration file (loaded at start)
        |     /etc/alertmanager/alertmanager.yml
        |     routing, receivers, inhibit, templates
        |     Git is the backup
        |
        +-- silences (in-memory, created via API or amtool)
        |     silence {id, matchers, startsAt, endsAt, createdBy, comment}
        |     the right backup is a periodic export to S3
        |
        +-- nflog (in-memory notification history)
              used for dedup across restarts
              ephemeral; the storage path can be set to disk
              but is rarely the right backup
        v
   Receivers (Slack, PagerDuty, email, webhook)

The backup shape:

   Stream 1: Git
   /etc/alertmanager/alertmanager.yml -> Git repository
   Git history = full configuration recovery.

   Stream 2: Silence export (every 5 minutes)
   GET /api/v2/silences -> JSON
   ship to s3://alertmanager-silences-*/
   the export is the recovery for in-flight silences.

   Stream 3 (optional): nflog persistence
   --storage.path=/var/lib/alertmanager
   nflog persists to disk on graceful shutdown.
   the right cadence is a clean shutdown, not a backup job.

A drill boots a staging Alertmanager with the Git configuration plus the most recent silence export, recreates the silences through the API, and confirms the staging instance accepts alerts from a test Prometheus.

How to configure it

The Alertmanager side — declare the configuration path and the storage path:

# /etc/alertmanager/alertmanager.yml
# SEVERITY: CONFIGURATION (reload)
global:
  resolve_timeout: 5m
  smtp_smarthost: 'smtp.internal:25'
  smtp_from: 'alertmanager@example.com'

route:
  receiver: 'default'
  group_by: ['alertname', 'cluster']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - severity = "page"
      receiver: 'pagerduty'
      continue: false
    - matchers:
        - severity = "ticket"
      receiver: 'slack-low'

receivers:
  - name: 'default'
    webhook_configs:
      - url: 'http://alertmanager-sink.internal:9099/'
  - name: 'pagerduty'
    pagerduty_configs:
      - service_key: '${PAGERDUTY_SERVICE_KEY}'
  - name: 'slack-low'
    slack_configs:
      - api_url: '${SLACK_WEBHOOK_URL}'
        channel: '#alerts-low'

inhibit_rules:
  - source_matchers:
      - severity = "page"
    target_matchers:
      - severity = "ticket"
    equal: ['alertname', 'cluster']

# The storage path is for the nflog and the silences persistence
# on graceful shutdown. The path is set on the command line, not
# in the YAML.
# --storage.path=/var/lib/alertmanager

The Alertmanager command line — declare the storage path:

# /etc/systemd/system/alertmanager.service
# SEVERITY: CONFIGURATION
[Unit]
Description=Alertmanager
After=network.target

[Service]
User=alertmanager
ExecStart=/usr/local/bin/alertmanager \
  --config.file=/etc/alertmanager/alertmanager.yml \
  --storage.path=/var/lib/alertmanager \
  --web.external-url=https://alertmanager.internal/
Restart=on-failure

[Install]
WantedBy=multi-user.target

The silence export — a periodic job:

#!/usr/bin/env bash
# SEVERITY: READ-ONLY
# The export is a defense-in-depth copy. The primary copy is the
# Alertmanager's own memory + storage path; the export catches
# the case where the Alertmanager cluster is wiped.
set -euo pipefail
AM_URL="http://alertmanager.internal:9093"
BUCKET="s3://alertmanager-silences-${AWS_REGION}"
STAMP="$(date -u +%Y-%m-%dT%H%M%SZ)"

# 1. List the active silences.
curl -fsS "${AM_URL}/api/v2/silences" \
  | jq '[.[] | select(.status.state == "active")]' \
  > "/tmp/silences-${STAMP}.json"

# 2. Ship to a versioned bucket.
AWS_PROFILE=am-backup aws s3 cp \
  --storage-class STANDARD \
  --sse aws:kms \
  --sse-kms-key-id "${AM_KMS_KEY}" \
  "/tmp/silences-${STAMP}.json" \
  "${BUCKET}/${STAMP}/silences.json"

rm -f "/tmp/silences-${STAMP}.json"

The silence recreate — the recovery side of the same shape:

#!/usr/bin/env bash
# SEVERITY: SERVICE-IMPACT (POSTs to a staging Alertmanager)
# Recreate the silences from the latest export.
set -euo pipefail
AM_URL="http://alertmanager-staging.internal:9093"
LATEST=$(AWS_PROFILE=am-backup aws s3 ls \
  s3://alertmanager-silences-${AWS_REGION}/ --recursive \
  | sort | tail -1 | awk '{print $4}')

AWS_PROFILE=am-backup aws s3 cp \
  "s3://alertmanager-silences-${AWS_REGION}/${LATEST}" - \
  | jq -c '.[]' | while read -r silence; do
    # Strip the id and status; the API assigns a new id.
    CREATE_BODY=$(echo "${silence}" | jq 'del(.id, .status, .updatedAt)')
    curl -fsS -X POST \
      -H 'Content-Type: application/json' \
      -d "${CREATE_BODY}" \
      "${AM_URL}/api/v2/silences" >/dev/null
  done

The configuration validation — amtool check-config is the right primitive:

# SEVERITY: READ-ONLY
amtool check-config /etc/alertmanager/alertmanager.yml
# Expected:
# Checking '/etc/alertmanager/alertmanager.yml'  SUCCESS
#
# Found:
#   - global config
#   - route
#   - 3 inhibit rules
#   - 5 receivers
#   - 1 templates

How to validate it

Top-level: the configuration matches Git and the silence export is fresh.

# SEVERITY: READ-ONLY
# The on-disk configuration should match the Git checkout.
diff -q /etc/alertmanager/alertmanager.yml \
  /opt/alertmanager-config/alertmanager.yml \
  || echo "DRIFT"

# The latest silence export should be less than 15 minutes old.
LATEST=$(AWS_PROFILE=am-backup aws s3 ls \
  s3://alertmanager-silences-${AWS_REGION}/ --recursive \
  | sort | tail -1 | awk '{print $1, $2}')
echo "Latest silence export: ${LATEST}"

Mid-level: the configuration is valid and the silence export parses.

# SEVERITY: READ-ONLY
amtool check-config /etc/alertmanager/alertmanager.yml

# Parse the export.
LATEST=$(AWS_PROFILE=am-backup aws s3 ls \
  s3://alertmanager-silences-${AWS_REGION}/ --recursive \
  | sort | tail -1 | awk '{print $4}')
AWS_PROFILE=am-backup aws s3 cp \
  "s3://alertmanager-silences-${AWS_REGION}/${LATEST}" - \
  | jq '. | length'
# Expected: an integer; the number of active silences at export time.

End-level: the drill produced a working Alertmanager.

# SEVERITY: READ-ONLY
cat /var/backups/alertmanager/drill/last-drill.txt
# Last successful restore drill of alertmanager: 2026-05-14,
# served a Git-loaded configuration plus 12 active silences
# recreated from the export.

The drill runbook in skeleton form:

# SEVERITY: SERVICE-IMPACT (boots a staging Alertmanager)
STAGE=/opt/am-drill
mkdir -p "${STAGE}/config" "${STAGE}/data"
# 1. Pull the configuration from Git.
git clone git@git.internal:alertmanager/config.git "${STAGE}/config"
# 2. Pull the latest silence export.
LATEST=$(AWS_PROFILE=am-backup aws s3 ls \
  s3://alertmanager-silences-${AWS_REGION}/ --recursive \
  | sort | tail -1 | awk '{print $4}')
AWS_PROFILE=am-backup aws s3 cp \
  "s3://alertmanager-silences-${AWS_REGION}/${LATEST}" "${STAGE}/silences.json"
# 3. Boot the staging Alertmanager.
docker run -d --name am-drill \
  -p 9094:9093 \
  -v "${STAGE}/config:/etc/alertmanager" \
  -v "${STAGE}/data:/alertmanager" \
  prom/alertmanager:v0.27.0 \
  --config.file=/etc/alertmanager/alertmanager.yml \
  --storage.path=/alertmanager
sleep 15
# 4. Recreate the silences.
bash recreate_silences.sh
# 5. Smoke test: query the silences API and confirm the count.
curl -s http://localhost:9094/api/v2/silences \
  | jq '[.[] | select(.status.state == "active")] | length'
# Expected: 12 (matching the export count).

How it can fail

Six failure modes recur in Alertmanager backup:

1. Silences are in memory only and the Alertmanager restarts. The on-call engineer applied a config reload that triggered a peer-state mismatch; the Alertmanager restarted; the silences were gone. Symptom: pages fire for alerts the team had silenced; the team scrambles to recreate the silences by hand.
2. The configuration drift between Git and disk. A team edited /etc/alertmanager/alertmanager.yml directly to fix a routing issue; the change was never committed. The next config reload picked up the Git version. Symptom: the routing is wrong; the team does not know which version is live.
3. The silence export has not run for a day. A cron job was disabled during a security exercise; the bucket has no new keys for 26 hours. Symptom: a real restart in that window loses the silences; the export is too stale to be useful.
4. The drill was never run. A quarterly drill on the calendar was scheduled and never executed. Symptom: RestoreDrillOverdue alert fires for a year; the team treats it as background noise.
5. The Alertmanager cluster split-brain. In HA mode the gossip protocol can briefly split; one peer has a silence the other does not. Symptom: an alert fires on one peer and not on the other; the page count is half of expected; the on-call engineer is confused.
6. The nflog was exhausted. The nflog has a fixed size; once it is full, older notifications are evicted. Symptom: an alert re-fires immediately after a notification because the nflog has dropped the dedup entry.

How to troubleshoot it

The order is: is the configuration in Git, are the silences exported, is the storage path set, does the staging Alertmanager serve the expected surface.

1. Is the configuration in Git? git status in the configuration repo. A dirty tree is a finding; reconcile by re-deploying from Git.
2. Are the silences exported? Check the bucket. Empty or stale means the export job is failing; investigate the cron service.
3. Is the storage path set? Check the command line. Without --storage.path, the nflog and the silence persistence are in memory; a crash loses them.
4. Does the staging Alertmanager serve the expected surface? Run the drill. If the drill has not run in the policy window, schedule it before any other change.
5. Are the receivers reachable? A recovered Alertmanager that cannot reach Slack or PagerDuty will queue notifications and eventually drop them. The drill must verify a real notification.

Security implications

• The Alertmanager configuration can hold receiver credentials (Slack webhook URLs, PagerDuty service keys). Treat the Git repository as sensitive. Use a secret manager for the credentials; do not commit them to Git.
• The silence export holds the (matchers, createdBy, comment) tuples. The comment field is free text and can include on-call engineer names and incident identifiers. Treat the export bucket as configuration data.
• KMS encryption with a customer-managed key is mandatory. The encryption key is rotated independently of the bucket.
• The receivers are integration points. A compromised receiver credential (a leaked Slack webhook) can be used to inject alerts. The credential rotation cadence matches the receiver SLA, not the bucket lifecycle.
• The drill staging host holds the configuration and the silence export. Treat the staging host as production data; wipe on completion.

Performance implications

• The silence export is small (a few KB per silence). The cost is negligible.
• The Git clone of the configuration is sub-second.
• A drill that boots a staging Alertmanager against the DR setup is a few hundred MB of disk and a few seconds of network. The staging host is small.
• The silence recreate script posts one POST per silence; for hundreds of silences this is sub-second.
• The nflog size is bounded. Setting --storage.path to a directory allows persistence but does not increase the size; the --storage.retention flag controls how long notifications stay in the log.

Production guidance

• Configuration in Git. The Git history is the primary backup. Validate with amtool check-config on every PR.
• Silences exported every 5-15 minutes via the API to a versioned bucket. KMS encryption with a customer-managed key.
• --storage.path set on the command line so the silence and nflog state persist across graceful restarts.
• Alert on silence-export staleness. The export must run; the schedule must be monitored.
• Restore drill quarterly. Smoke test: load the Git configuration, recreate the silences from the export, send a test alert, confirm a notification reaches a test receiver.
• Alert on Alertmanager restart. A restart wipes in-memory state; the alert catches the restart before the team notices the silence loss.
• The backup IAM role cannot delete. Lifecycle handles expiry.
• For HA, monitor gossip state. A split-brain is a recovery problem before it is a notification problem.

Verification

You should now be able to answer:

• What are the three Alertmanager states, and which need a backup?
• Why is the silence export periodic rather than continuous, and why does it matter?
• What is the right primitive for taking a configuration backup, and what is the right primitive for taking a silence backup?
• Why does a graceful restart preserve silences when --storage.path is set, but a crash does not?
• What is the dual smoke test for an Alertmanager restore drill?

Quiz