Grouping of Alerts

A database host starts to fill its disk. Prometheus fires DiskFillingSoon on the host. Then the WAL partitions fill. PostgresWALSpaceLow fires. Then a query backlog appears. PostgresBacklogGrowing fires. Three alerts, one host, one incident.

A well-grouped Alertmanager configuration collapses these into a single notification: “host X has a disk incident.” A poorly-grouped configuration sends three separate Slack messages and three separate PagerDuty pages. The on-call engineer reads all three and recognises the same incident. Multiply by a fleet of fifty hosts.

Grouping is the difference between one page per incident and one page per symptom.

What it is

Grouping is the mechanism by which Alertmanager decides which firing alerts are sent together in a single notification. It is controlled by two related settings:

• group_by: — the labels whose combined value determines the group key. Two alerts with the same group_by values share a group. Two alerts with different group_by values do not.
• group_wait: — the time the dispatcher waits for more alerts to arrive before sending the first notification for a new group. Lets a wave of correlated alerts collapse into one page.
• group_interval: — the time the dispatcher waits between subsequent notifications for the same group, when new alerts keep arriving or existing alerts update.
• repeat_interval: — the minimum time between reminder notifications for the same still-firing group, even when nothing has changed.

The default grouping key is alertname alone. In production that default is rarely what you want.

Why a sysadmin cares

The two ends of the grouping spectrum each cost real money.

Too coarse (group on too few labels, e.g. alertname alone): one PagerDuty page acknowledges every firing instance of the same alert across the entire fleet. The on-call resolves the PagerDuty incident the moment one host is fixed. The other hosts stay broken silently. Worse: a single PagerDuty incident covers an unrelated set of root causes — a HostDown on a switch and a HostDown on a Kubernetes node show up as one incident because they share an alertname.

Too fine (group on high-cardinality labels, e.g. instance): every host pages independently. The same incident becomes fifty pages. The on-call team silences the alert out of fatigue. A real emergency finds a silenced team.

The right granularity is the boundary that defines one incident. For most rules, that boundary is alertname + cluster + service, or alertname + alertgroup. For service-level alerts it might be alertname + team + service. The grouping key is a product decision, not an implementation detail.

How it works

The grouping state lives in nflog/. When an alert arrives, its labels are intersected with the group_by list. The resulting tuple is the group key. AM then checks whether a group with that key already exists. If yes, the alert joins it. If no, a new group is created.

  alert A: alertname=DiskFillingSoon cluster=prod-eu-1 severity=warning
  alert B: alertname=PostgresWALSpaceLow cluster=prod-eu-1 severity=warning
  alert C: alertname=PostgresBacklogGrowing cluster=prod-eu-1 severity=warning

  group_by: ['alertname', 'cluster']

  group keys produced:
    A: (DiskFillingSoon, prod-eu-1)
    B: (PostgresWALSpaceLow, prod-eu-1)
    C: (PostgresBacklogGrowing, prod-eu-1)

  groups in nflog: 3 (one per alertname)
  notifications sent: 3

Now change group_by to ['cluster', 'severity']:

  group keys produced:
    A: (prod-eu-1, warning)
    B: (prod-eu-1, warning)
    C: (prod-eu-1, warning)

  groups in nflog: 1
  notifications sent: 1 (after group_wait)

The same three alerts, one notification, because the grouping key is now “the cluster is in warning.” The recipient sees all three alerts in a single Slack message and reads them in the context of the cluster.

Under the hood

Groups are stored in nflog/. The retention window is bounded by --data.retention and the default window for the nflog notification decision. Once an alert resolves, AM removes it from the group. The group itself is removed when the last alert resolves.

The three timers operate independently:

  T0: alert A fires
  T0: group created (group key K)
  T0: group_wait timer starts (30s default)

  T0+5s: alert B fires
  T0+5s: B joins group K
  T0+5s: group_wait timer continues

  T0+12s: alert C fires
  T0+12s: C joins group K

  T0+30s: group_wait expires
  T0+30s: notification 1 sent (alerts A, B, C)
  T0+30s: group_interval timer starts (5m default)

  T1+10s: alert D fires
  T1+10s: D joins group K
  T1+10s: group_interval continues

  T5+30s: group_interval expires (no new alerts in window)
  T5+30s: repeat_interval timer starts (4h default)

  T5h+30s: repeat_interval expires
  T5h+30s: notification 2 sent (still-firing alerts)

The interaction matters. A group_interval shorter than repeat_interval would create “I am about to repeat” notifications that arrive before the repeat window. Make repeat_interval at least an order of magnitude longer than group_interval.

How to configure it

A reasonable starting point for a multi-cluster fleet:

route:
  receiver: 'default-slack'
  group_by: ['alertname', 'cluster']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    # Critical alerts: tighter grouping, faster timers, more repetition.
    - matchers:
        - severity = critical
      receiver: 'pagerduty-oncall'
      group_by: ['alertname', 'cluster', 'service']
      group_wait: 10s
      group_interval: 2m
      repeat_interval: 1h
    # Info alerts: loose grouping, slow timers, no repetition.
    - matchers:
        - severity = info
      receiver: 'slack-audit'
      group_by: ['alertname']
      group_wait: 1m
      group_interval: 30m
      repeat_interval: 24h

Three knobs to reason about:

• group_by: the labels whose value defines “one incident”. Always include alertname. Add labels that distinguish independent incidents (cluster, service, team).
• group_wait: how long to wait for the wave. 30s is a reasonable default; longer for batch jobs that emit alerts in clusters.
• group_interval: the rate of updates while the incident is still evolving. 5m is reasonable for warnings; tighten for critical alerts where new information is worth a fresh notification.
• repeat_interval: the rate of reminders for a stable firing group. Critical alerts should repeat inside the on-call SLA. Info alerts can be “until resolved” (repeat_interval: 24h or longer).

How to validate it

amtool config routes test shows the grouping state for a synthetic alert. Combine it with the live API to confirm what the running instance is actually doing.

# What group does this alert join?
amtool config routes test \
  --config.file=/etc/alertmanager/alertmanager.yml \
  alertname=DiskFillingSoon cluster=prod-eu-1 severity=warning
# Output:
# Selected receiver: default-slack
# Grouping:
#   alertname: DiskFillingSoon
#   cluster: prod-eu-1
# Group wait: 30s
# Group interval: 5m
# Repeat interval: 4h

Now look at what the running instance is doing:

# Currently active alerts, grouped by receiver.
curl -s http://localhost:9093/api/v2/alerts \
  | jq -r '.[] | "\(.labels.alertname)/\(.labels.cluster) -> \(.receivers[0].name)"' \
  | sort | uniq -c
# Output (illustrative):
#    3 DiskFillingSoon/prod-eu-1 -> slack-default
#    2 HighRequestLatency/prod-us-2 -> slack-default

The shape of the output is the shape of your incidents. Three alerts in one group is a single incident; the grouping key is doing its job.

For each unique receiver, you can also see the per-group timer state:

curl -s http://localhost:9093/api/v2/alerts \
  | jq '.[] | {labels: .labels, receivers: .receivers[].name, status: .status}'

status.state will be active or suppressed. suppressed alerts are inside a group whose notification window has not yet elapsed.

How it can fail

The recurring failure modes, in descending order of operational cost:

1. Grouping on a high-cardinality label. A team adds container_id to group_by. Every replica of a service produces its own group. A 30-replica deployment now sends 30 pages for one underlying issue. The fix is to remove the label from group_by and rely on the parent grouping key.
2. Grouping too coarsely for a heterogeneous incident. A group_by: ['alertname'] collapses a HostDown on the database cluster with a HostDown on the edge router into a single PagerDuty incident. The on-call resolves one and the other stays open. The fix is to add cluster or region to the grouping key.
3. group_wait too short. A batch job fires 50 alerts at once. group_wait: 5s means the first notification goes out before the batch has finished firing. The recipient sees a “partial” incident; the next notification arrives 90 seconds later as an update. Tune group_wait to the typical wave duration of the alert sources.
4. repeat_interval shorter than group_interval. The two timers interact: AM will fire a repeat before the group_interval fires, creating a notification stream that looks like a runaway loop. Set repeat_interval to at least 5x group_interval.
5. group_by: ['...'] does not include alertname. Two unrelated alerts with identical remaining labels collapse into a single notification. The recipient gets a confusing “alertname=HostDown AND alertname=DiskFillingSoon” notification. Always include alertname in group_by.
6. Per-route group_by drift. A child route inherits the parent’s group_by but the inheriting branch needs a different granularity. The override is missing; groups are coarser than intended. Always set group_by explicitly on the branches where it matters.

How to troubleshoot it

When the grouping is wrong, the symptom is either too many notifications (too fine) or too few (too coarse). The diagnostic order:

1. Pull the live grouping state. curl -s http://localhost:9093/api/v2/alerts | jq shows the raw labels and the receiver per alert.
2. Recompute the group key by hand. Strip the alert labels down to the group_by list. The remaining tuple is the group key. Compare against the actual notification grouping in the AM log.
3. Tail the AM log for aggrGroup=.... Each notification line includes the group key. If the key is wrong, the group_by list is wrong.
4. Run amtool config routes test with the exact labels. The output shows the grouping that would apply for a synthetic alert. If the synthetic test is right and the live state is wrong, you have a SIGHUP failure (the running AM has the old config loaded).

Security implications

Grouping is not security-sensitive on its own. Two adjacent risks:

• Information exposure in grouped notifications. When a group contains alerts from different owners (e.g. an SRE alert and a security alert collapsed into one notification), the recipient sees both. The grouping boundary is a disclosure boundary; align it with the information-classification boundary.
• Receiver load as a denial-of-service surface. A receiver that throttles AM (a Slack webhook at the workspace limit) is also throttling every other group that uses the same receiver. Grouping granularity directly controls the rate.

Performance implications

The hot path is the group-key hash lookup on every alert. AM keeps the active groups in memory; the cost is O(1) per alert. The bottleneck is downstream:

• Notification log size per group. Each notification attempt is an append-only file under notify/<receiver-hash>/. A high-cardinality group_by produces many small files; this is fine. A low-cardinality group_by with frequent updates produces fewer but larger files; this is also fine. The failure shape is a single very chatty group on a slow receiver — the file grows, the receiver times out, the retry accumulates.

• AM goroutines per receiver. One per receiver, not one per group. Grouping granularity does not affect the goroutine count. The receiver timeout (timeout: in the receiver config) is the real lever.

Capacity planning: a single AM node on 1 vCPU / 1 GiB RAM handles tens of thousands of groups in nflog without degradation. The receiver-side rate limits and the storage path are the limits.

Production guidance

• Always include alertname in group_by. Two unrelated alerts with the same other labels should never collapse.
• Group by the label that defines “one incident”. For most rules, cluster or service. For cross-cluster incidents, team. For service-level alerts, the service name.
• Tune timers per severity. Critical alerts repeat inside the SLA; warnings update every few minutes; info alerts can wait an hour.
• Audit the live group state. A weekly pull of GET /api/v2/alerts into a CSV, grouped by receiver, is the cheapest way to catch grouping drift before it becomes a paging incident.

Verification

You should now be able to answer:

• What is the difference between group_by, group_wait, group_interval, and repeat_interval?
• What happens when two alerts share the same group_by values?
• Why is grouping on a high-cardinality label a production failure mode?
• What is the right granularity for “one incident” in a typical multi-cluster fleet?
• How do you inspect the live group state of a running Alertmanager?

Quiz