Inhibit, then Page

Inhibition is the dry-run of escalation. Severity is the wet-run. A warning alert fires when the latency crosses 1 s; a critical alert fires when it crosses 5 s. Both share the same alertname. While the critical alert is active, the warning alert is suppressed — the operator should only be paged for the critical. When the critical alert resolves, the warning alert (if still firing) re-appears and is paged again, because the underlying condition is still degraded but not catastrophic.

This is the discipline of inhibit, then page: let the critical alert drive paging, let the warning alert stand by as a fallback. The two disciplines — inhibition and routing — must be designed together. A route that pages every warning alert defeats the inhibit rule. An inhibit rule that suppresses the critical alert leaves the operator with nothing.

What it is

The pattern is two-tier escalation:

• A warning severity alert fires when a condition is degraded but not catastrophic.
• A critical severity alert fires when the same condition becomes catastrophic.
• An inhibit rule suppresses the warning alert whenever the critical alert is firing on the same labels.
• A route sends the critical alert to the pager; a separate route sends the warning alert to a ticket queue.

Time 0:00  ServiceLatencyHigh fires at severity=warning
          -> route to "ticket-queue" receiver
          -> warning page in the ticket system

Time 0:05  ServiceLatencyHigh fires at severity=critical
          -> inhibit rule suppresses the warning alert
          -> route to "pager" receiver
          -> critical page to on-call

Time 0:30  ServiceLatencyHigh resolves at severity=critical
          -> inhibit rule no longer applies
          -> ServiceLatencyHigh still firing at severity=warning
          -> route to "ticket-queue" receiver
          -> warning page in the ticket system (re-fires)

Time 1:00  ServiceLatencyHigh resolves at severity=warning
          -> no active alerts
          -> no notification

The warning alert acts as the “still degraded but not paging” fallback. The critical alert drives the page. When critical resolves, the warning is allowed to surface again because the underlying problem is still there.

Why a sysadmin cares

The single biggest source of pager fatigue in a mature platform is the team that pages every warning alert. Every warning alert that crosses the pager threshold trains the on-call to ignore the next one. Within months, the team is asleep for real incidents because the pager is a “warning soup” of background radiation.

Inhibition, paired with severity-tiered routing, is the mitigation. The warning alert stays in the system as a ticket. The critical alert drives the page. When the critical alert fires, the warning alert is suppressed and the operator is paged for the right thing.

How it works

There are three distinct pieces:

Piece 1: the route tree

route:
  receiver: default
  group_by: [alertname, instance]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    # Critical alerts page the on-call.
    - matchers:
        - severity="critical"
      receiver: pager
      group_wait: 10s
      repeat_interval: 1h
    # Warning alerts go to the ticket queue.
    - matchers:
        - severity="warning"
      receiver: ticket-queue
      group_wait: 1m
      repeat_interval: 24h

Two child routes match on severity. The first matches severity=critical and routes to the pager. The second matches severity=warning and routes to the ticket queue. The matchers are mutually exclusive; an alert matches one or the other.

Piece 2: the inhibit rule

inhibit_rules:
  - source_matchers:
      - alertname="ServiceLatencyHigh"
      - severity="critical"
    target_matchers:
      - alertname="ServiceLatencyHigh"
      - severity="warning"
    equal: [instance, service]

The rule suppresses the warning alert whenever the critical alert is firing on the same instance and service. The source and target share the alertname; only the severity differs.

Piece 3: the receivers

receivers:
  - name: default
    webhook_configs:
      - url: 'http://localhost:5001/alerts'
  - name: pager
    pagerduty_configs:
      - service_key: '<pagerduty-key>'
  - name: ticket-queue
    webhook_configs:
      - url: 'http://localhost:5002/ticket'

Three receivers. The default receiver is the fallback. The pager receiver pages the on-call via PagerDuty (or Opsgenie, or VictorOps). The ticket-queue receiver opens a ticket without paging anyone.

The merged pipeline

Source alert :  ServiceLatencyHigh severity=critical
Target alert :  ServiceLatencyHigh severity=warning

Step 1: Group by [alertname, instance]
        -> one group for ServiceLatencyHigh

Step 2: Inhibit
        -> critical alert is the source
        -> warning alert is the target
        -> equal [instance, service] is satisfied
        -> warning alert is marked suppressed

Step 3: Route
        -> critical alert matches severity="critical" -> pager
        -> warning alert is suppressed, not routed
        -> critical alert pages; warning alert is silent

Step 4: Notify
        -> PagerDuty receives the critical alert
        -> Ticket queue receives nothing for the warning

The two pieces — route and inhibit — work together. The route decides that critical pages. The inhibit rule prevents the warning from also paging. The operator gets one page for one condition.

Under the hood

How to configure it

A complete configuration showing the merge of routes and inhibitions for a service that has both warning and critical severity tiers.

# /etc/alertmanager/alertmanager.yml
route:
  receiver: default
  group_by: [alertname, cluster, instance]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    # Critical alerts page the on-call immediately.
    - matchers:
        - severity="critical"
        - env="prod"
      receiver: pager
      group_wait: 10s
      group_interval: 1m
      repeat_interval: 1h
      continue: false
    # Warning alerts go to the ticket queue.
    - matchers:
        - severity="warning"
        - env="prod"
      receiver: ticket-queue
      group_wait: 1m
      group_interval: 5m
      repeat_interval: 24h
      continue: false

inhibit_rules:
  # Critical alert suppresses the warning alert on the same
  # instance and service. The operator pages once.
  - source_matchers:
      - alertname=~"^ServiceLatencyHigh$"
      - severity="critical"
    target_matchers:
      - alertname=~"^ServiceLatencyHigh$"
      - severity="warning"
    equal: [instance, service, cluster]

receivers:
  - name: default
    webhook_configs:
      - url: 'http://localhost:5001/alerts'

  - name: pager
    pagerduty_configs:
      - service_key: '<pagerduty-service-key>'
        send_resolved: true

  - name: ticket-queue
    webhook_configs:
      - url: 'http://ticket-system.example.local/alerts'
        send_resolved: true

Four things to read carefully:

1. continue: false on every child route. When the first child matches, evaluation stops. The default receiver is the fallback for alerts that match no child.
2. send_resolved: true on the pager receiver. When the critical alert resolves, PagerDuty receives a resolved event and the incident closes. Without this, the PagerDuty incident stays open until manually closed.
3. equal: [instance, service, cluster]. Three labels constrain the suppression. A critical alert on prod-eu/checkout-svc/prod-checkout-12 suppresses the warning on the same combination. A critical alert on prod-us does not suppress a warning on prod-eu.
4. The route’s matchers and the inhibit rule’s source_matchers use the same label vocabulary. Both match on severity=critical. The label is the bridge between the two.

How to validate it

Three checks. The third is the escalation dry-run.

# 1. Static syntax check (CONFIGURATION severity)
amtool check-config /etc/alertmanager/alertmanager.yml

Expected output (illustrative):

Checking /etc/alertmanager/alertmanager.yml
  SUCCESS
Parsed and loaded successfully
Found route 'critical' with receiver 'pager'
Found route 'warning' with receiver 'ticket-queue'
Found inhibit rule 'Critical-Suppresses-Warning-Latency'

# 2. SIGHUP reload (CONFIGURATION severity)
kill -HUP "$(pidof alertmanager)"

# 3. Escalation dry-run (READ-ONLY)
#    Phase A — only the warning alert fires.
amtool alert add alertname=ServiceLatencyHigh \
  instance="prod-checkout-12" service="checkout-svc" \
  cluster="prod-eu" env="prod" severity="warning"

sleep 2
echo "Phase A — warning only:"
curl -s 'http://localhost:9093/api/v2/alerts?active=true&silenced=true' \
  | jq '.[] | {severity: .labels.severity,
                status: .status.state,
                receivers: .receivers[].name}'

#    Phase B — the critical alert fires. The warning should
#    be suppressed.
amtool alert add alertname=ServiceLatencyHigh \
  instance="prod-checkout-12" service="checkout-svc" \
  cluster="prod-eu" env="prod" severity="critical"

sleep 2
echo "Phase B — warning + critical:"
curl -s 'http://localhost:9093/api/v2/alerts?active=true&silenced=true' \
  | jq '.[] | {severity: .labels.severity,
                status: .status.state,
                inhibitedBy: .status.inhibitedBy,
                receivers: .receivers[].name}'

#    Phase C — resolve the critical alert by re-posting the
#    same label set with a start and an end in the past.
#    amtool has no command that deletes an alert; alerts
#    leave Alertmanager by resolving. The warning should
#    re-appear.
amtool alert add alertname=ServiceLatencyHigh \
  instance="prod-checkout-12" service="checkout-svc" \
  cluster="prod-eu" env="prod" severity="critical" \
  --start="$(date -u -d '-5 minutes' +%Y-%m-%dT%H:%M:%SZ)" \
  --end="$(date -u -d '-1 minute' +%Y-%m-%dT%H:%M:%SZ)"

sleep 2
echo "Phase C — critical resolved, warning remains:"
curl -s 'http://localhost:9093/api/v2/alerts?active=true&silenced=true' \
  | jq '.[] | {severity: .labels.severity,
                status: .status.state,
                receivers: .receivers[].name}'

Expected output (illustrative):

Phase A:
{ "severity": "warning",
  "status": "active",
  "receivers": ["ticket-queue"] }

Phase B:
{ "severity": "warning",
  "status": "suppressed",
  "inhibitedBy": ["ServiceLatencyHigh/critical"],
  "receivers": [] }
{ "severity": "critical",
  "status": "active",
  "inhibitedBy": [],
  "receivers": ["pager"] }

Phase C:
{ "severity": "warning",
  "status": "active",
  "receivers": ["ticket-queue"] }

Phase A — warning is routed to ticket-queue. Phase B — critical is routed to pager; warning is suppressed. Phase C — critical resolves; warning re-appears and is routed to ticket-queue.

Clean up: the critical alert was already resolved in phase C, so only the warning is left. Resolve it the same way.

START="$(date -u -d '-5 minutes' +%Y-%m-%dT%H:%M:%SZ)"
END="$(date -u -d '-1 minute' +%Y-%m-%dT%H:%M:%SZ)"
amtool alert add alertname=ServiceLatencyHigh \
  instance="prod-checkout-12" service="checkout-svc" \
  cluster="prod-eu" env="prod" severity="warning" \
  --start="$START" --end="$END"

amtool alert query alertname=ServiceLatencyHigh

How it can fail

Five failure modes specific to the inhibit-then-page merge:

1. Route matches warning before inhibit suppresses it. The route tree evaluates before inhibition in some legacy Alertmanager versions; in 0.28.x the order is fixed (inhibit then route). Symptom: warning alert pages the operator before being suppressed. Verify with the dry-run above; if the order is wrong, upgrade Alertmanager.
2. Severity label is missing on the source. The critical alert does not have severity=critical in its labels. The inhibit rule’s source matcher does not fire. Symptom: warning alert is not suppressed. The route still sends the warning to the ticket queue, but the operator is also paged for the warning. Investigation: query the source alert; confirm .labels.severity.
3. equal: labels disagree between source and target. The warning alert has cluster="prod_eu" (underscore) and the critical has cluster="prod-eu" (hyphen). Symptom: the suppress does not fire. Investigation: diff every label in equal: byte-for-byte.
4. Catch-all route shadowing the severity routes. A child route with no matchers is listed above the severity routes. Symptom: every alert matches the catch-all first; the severity routes never fire. Investigation: read the route tree top-down; confirm the catch-all (if any) is below the severity routes.
5. send_resolved: false on the pager receiver. Symptom: when the critical alert resolves, the PagerDuty incident stays open. The on-call is paged again on the next firing because the incident from the previous cycle is still active. Investigation: check the receiver config; set send_resolved: true.

How to troubleshoot it

The diagnostic order when the escalation does not behave as expected.

1. Is the critical alert firing with the right severity? curl /api/v2/alerts?active=true&filter=severity=critical. Confirm that .labels.severity reads critical. A typo or missing label means the route and the inhibit rule both fail to match.
2. Is the warning alert suppressed? curl /api/v2/alerts?silenced=true. The .status.inhibitedBy field should list the critical alert. An empty list means the inhibit rule did not fire.
3. Is the route tree evaluated correctly? Read the route tree top-down. Confirm the order: catch-all should be below severity routes. Use the API’s /api/v2/status to inspect the loaded route tree.
4. Is the receiver reachable? Check the receiver’s webhook or PagerDuty integration. A 500 from the receiver means the alert is sent but the page never arrives. The Alertmanager log shows the receiver’s response.
5. Read the alertmanager log. Run with --log.level=debug. The lines around inhibit, route, and notify show the full pipeline for each alert.

Security implications

The pager receiver carries credentials (PagerDuty service key, Opsgenie API key, Slack webhook URL). Treat alertmanager.yml as a credential-bearing file. Restrict repository access; do not commit the file to a public repository; use a secrets manager for the keys.

The route tree exposes the paging hierarchy — who is paged, when, and for what. A leaked route tree is a social engineering target. An attacker who knows the paging hierarchy can construct an alert that pages the right person at the wrong time.

Performance implications

The cost of the inhibit-then-page merge is constant per evaluation. The new operational cost is the maintenance of the two-tier severity discipline. Every alert that is expected to escalate must carry both a warning and a critical version. Every alert that does not have a critical version cannot be escalated; the operator pages for the warning or does not page at all.

Production guidance

• Every alert that pages must have a clear escalation path: warning → critical, or warning → ticket-only.
• The route tree’s severity routes must be above any catch-all. The order of children matters.
• The inhibit rule’s source must be the alert with the higher severity. The target must be the alert with the lower severity. The direction of the rule is not negotiable.
• Use send_resolved: true on the pager receiver. The PagerDuty incident should close when the alert resolves.
• Validate the escalation end-to-end after every change to either the route tree or the inhibit rule. The dry-run takes 30 seconds.

Verification

You should now be able to answer:

• What is the role of the route tree in the inhibit-then- page pattern?
• Why must the source of the inhibit rule be the higher- severity alert, and the target the lower-severity?
• How does the operator know whether the rule is firing at each phase of the escalation?
• What is the failure shape when a catch-all route shadows the severity routes?

Quiz