Inhibit by Cluster

A real production incident. On 03 August 2025 at 11:42 UTC, a staging cluster autoscaler triggered a reboot of stg-checkout-07. The host-down alert fired on the staging cluster. Because the inhibit rule for host-down used the same alertname across clusters and lacked a cluster filter, the production ServiceDown alert on prod-checkout-12 was also suppressed — not because the host was down, but because the staging alert matched the rule’s pattern. The production service was degraded for nineteen minutes before the on-call engineer noticed the suppression in the API and traced it back to the staging cluster. The post-mortem identified three things: the rule had no cluster equal clause; the rule had no region filter; and the staging/production separator was missing from the rule design.

This lesson teaches the discipline of cluster-aware inhibition: how to use cluster and region labels to scope every rule to the right tenancy, how to write a staging/ production separator, and how to validate that a rule’s blast radius does not leak across clusters.

What it is

Cluster-aware inhibition is the practice of carrying a cluster label (and usually a region label) on every alert, and writing every inhibit rule so that its blast radius is explicit. The rule either scopes to a single cluster, scopes to a region that contains multiple clusters, or — and this is rare and dangerous — scopes to every cluster globally.

Clusters in a typical platform:

  prod-eu       production traffic, EU customers
  prod-us       production traffic, US customers
  prod-ap       production traffic, APAC customers
  stg-eu        staging, mirrors prod-eu
  stg-us        staging, mirrors prod-us
  dev           developer sandboxes

Labels on every alert:

  cluster="prod-eu"   region="eu"   env="prod"
  cluster="stg-eu"    region="eu"   env="stg"
  cluster="dev"                       env="dev"

The cluster label identifies the tenancy. The region label identifies the geography. The env label is the coarse classification (production, staging, development). A well-designed inhibit rule uses all three labels to constrain its blast radius.

Why a sysadmin cares

Most platforms run more than one cluster. A typical mid-size platform runs production in three regions plus staging and development. A rule that does not scope by cluster will fire on staging hosts and suppress production alerts with the same alertname. The failure mode is silent because the suppression looks identical to a “real” suppression; only the cluster label on the source reveals the difference.

Two patterns are common:

• Staging/production separator. A single rule suppresses service-down alerts whenever a host-down alert is firing on the same cluster. The rule must constrain to the same cluster; otherwise, a staging host-down suppresses production service-down alerts.
• Region-wide upstream. A regional upstream (a load balancer, a database primary) goes down. Every cluster in the region should be suppressed; clusters in other regions should be unaffected. The rule must use region (or a finer label) in the equal clause to allow cross-cluster suppression within a region.

How it works

The matcher primitives are the same as the basic rule. The discipline is the labels.

Pattern 1 — single-cluster rule

- source_matchers:
    - alertname=~"^(HostDown|NodeDown)$"
    - severity="critical"
  target_matchers:
    - alertname=~"^ServiceDown$"
    - severity=~"warning|critical"
  equal: [cluster, instance]

The cluster label in equal: ensures the source and target are on the same cluster. A staging host-down cannot suppress a production service-down because their cluster labels differ.

Pattern 2 — region-wide rule

- source_matchers:
    - service="regional-lb-eu"
    - severity="critical"
  target_matchers:
    - depends_on="regional-lb-eu"
    - severity=~"warning|critical"
  equal: [region]

The region label in equal: allows the rule to suppress across clusters within the same region. The regional-lb-eu going down affects every cluster in eu, including prod-eu and stg-eu. The rule fires for both.

Pattern 3 — staging/production separator

- source_matchers:
    - alertname=~"^(HostDown|NodeDown)$"
    - severity="critical"
    - env="prod"
  target_matchers:
    - alertname=~"^ServiceDown$"
    - severity=~"warning|critical"
    - env="prod"
  equal: [cluster, instance]

The source matcher restricts to env="prod". A staging host-down cannot trigger the rule because its env label is stg. The target matcher also restricts to env="prod", so production alerts on the same cluster and instance are suppressed, while staging alerts are unaffected.

Pattern 4 — global rule (rare)

- source_matchers:
    - alertname="GlobalAuthDown"
    - severity="critical"
  target_matchers:
    - depends_on_team="auth"
    - severity=~"warning|critical"

No equal: clause. The rule fires globally. This is appropriate only for true global failures (an authentication service that every region and every cluster depends on). The default assumption is that the rule should be scoped; a global rule is the exception that requires an explicit comment.

Under the hood

How to configure it

A complete configuration showing all three patterns in one file. Note the comments on each rule — every rule’s blast radius is explicit.

# /etc/alertmanager/alertmanager.yml
route:
  receiver: default
  group_by: [cluster, alertname, instance]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

inhibit_rules:
  # 1. Single-cluster host rule. Default scope.
  #    Staging host-down does not suppress production service-down.
  - source_matchers:
      - alertname=~"^(HostDown|NodeDown|InstanceDown)$"
      - severity="critical"
      - env="prod"
    target_matchers:
      - alertname=~"^(ServiceDown|ExporterDown|ScrapeFailed)$"
      - severity=~"warning|critical"
      - env="prod"
    equal: [cluster, instance]

  # 2. Same rule, scoped to staging. Staging host-down does not
  #    suppress production at all; it suppresses staging only.
  - source_matchers:
      - alertname=~"^(HostDown|NodeDown|InstanceDown)$"
      - severity="critical"
      - env="stg"
    target_matchers:
      - alertname=~"^(ServiceDown|ExporterDown|ScrapeFailed)$"
      - severity=~"warning|critical"
      - env="stg"
    equal: [cluster, instance]

  # 3. Region-wide upstream rule. A regional load balancer
  #    outage suppresses every dependent in the region, across
  #    clusters.
  - source_matchers:
      - service="regional-lb-eu"
      - severity="critical"
    target_matchers:
      - depends_on="regional-lb-eu"
      - severity=~"warning|critical"
    equal: [region]

  # 4. Global auth rule. GLOBAL blast radius — explicit comment.
  #    This rule applies to every cluster and region because
  #    the auth service is a global dependency. Review quarterly.
  - source_matchers:
      - alertname="AuthServiceDown"
      - severity="critical"
    target_matchers:
      - depends_on_team="auth"
      - severity=~"warning|critical"
    # NOTE: no equal: clause is intentional for global
    # dependencies. Any change here requires an SRE review.

receivers:
  - name: default
    webhook_configs:
      - url: 'http://localhost:5001/alerts'

Three things to read carefully:

1. Pattern 1 vs Pattern 2. The same rule exists for env="prod" and env="stg". The rules are duplicated rather than parameterised because Alertmanager does not support rule templates. The duplication is intentional; it makes every rule’s scope explicit.
2. Pattern 3 has no cluster in equal:. The rule is scoped to a region, not a cluster. The region label allows the rule to fire across clusters within the region.
3. Pattern 4 is global. The comment is required. A reviewer reading the rule six months later needs to know why the blast radius is the entire platform.

How to validate it

Four checks, including a staging/production separator test that proves the staging/production boundary holds.

# 1. Static syntax check (CONFIGURATION severity)
amtool check-config /etc/alertmanager/alertmanager.yml

Expected output (illustrative):

Checking /etc/alertmanager/alertmanager.yml
  SUCCESS
Parsed and loaded successfully
Found inhibit rule 'HostDown-Suppresses-ServiceDown-Prod'
Found inhibit rule 'HostDown-Suppresses-ServiceDown-Stg'
Found inhibit rule 'RegionalLBDown-Suppresses-Dependents'
Found inhibit rule 'AuthServiceDown-Suppresses-Dependents'

# 2. SIGHUP reload (CONFIGURATION severity)
kill -HUP "$(pidof alertmanager)"

# 3. Loaded-config echo (READ-ONLY)
# The status API returns the running configuration as a YAML
# string in .config.original, not as a parsed object.
curl -s http://localhost:9093/api/v2/status \
  | jq -r '.config.original' | grep -A8 '^inhibit_rules:'

# 4. Staging/production separator dry-run (READ-ONLY)
#    A staging host-down should NOT suppress a production
#    service-down. The dry-run proves the separator works.
amtool alert add alertname=HostDown \
  cluster="stg-eu" env="stg" \
  instance="stg-checkout-07" severity="critical"

amtool alert add alertname=ServiceDown \
  cluster="prod-eu" env="prod" \
  instance="prod-checkout-12" severity="warning"

sleep 2
curl -s 'http://localhost:9093/api/v2/alerts?active=true&silenced=true' \
  | jq '.[] | select(.labels.alertname=="ServiceDown"
                     and .labels.cluster=="prod-eu")
         | {status: .status.state,
            inhibitedBy: .status.inhibitedBy}'

Expected output: status: "active", inhibitedBy: []. The production alert is NOT suppressed. The staging host-down does not leak across the staging/production boundary.

amtool has no command that deletes an alert. Alerts leave Alertmanager by resolving, not by being removed: an alert posted without an explicit end time resolves on its own after the global resolve_timeout (five minutes by default). To resolve one immediately, re-post the same label set with a start and an end that are both in the past.

# Clean up: resolve the two synthetic alerts.
START="$(date -u -d '-5 minutes' +%Y-%m-%dT%H:%M:%SZ)"
END="$(date -u -d '-1 minute' +%Y-%m-%dT%H:%M:%SZ)"
amtool alert add alertname=HostDown \
  cluster="stg-eu" env="stg" \
  instance="stg-checkout-07" severity="critical" \
  --start="$START" --end="$END"
amtool alert add alertname=ServiceDown \
  cluster="prod-eu" env="prod" \
  instance="prod-checkout-12" severity="warning" \
  --start="$START" --end="$END"

The positive case — a production host-down DOES suppress a production service-down on the same cluster and instance:

amtool alert add alertname=HostDown \
  cluster="prod-eu" env="prod" \
  instance="prod-checkout-12" severity="critical"

amtool alert add alertname=ServiceDown \
  cluster="prod-eu" env="prod" \
  instance="prod-checkout-12" severity="warning"

sleep 2
curl -s 'http://localhost:9093/api/v2/alerts?active=true&silenced=true' \
  | jq '.[] | select(.labels.alertname=="ServiceDown"
                     and .labels.cluster=="prod-eu")
         | {status: .status.state,
            inhibitedBy: .status.inhibitedBy}'

Expected output: status: "suppressed", inhibitedBy: ["HostDown"]. The production alert IS suppressed. Resolve this pair the same way:

START="$(date -u -d '-5 minutes' +%Y-%m-%dT%H:%M:%SZ)"
END="$(date -u -d '-1 minute' +%Y-%m-%dT%H:%M:%SZ)"
amtool alert add alertname=HostDown \
  cluster="prod-eu" env="prod" \
  instance="prod-checkout-12" severity="critical" \
  --start="$START" --end="$END"
amtool alert add alertname=ServiceDown \
  cluster="prod-eu" env="prod" \
  instance="prod-checkout-12" severity="warning" \
  --start="$START" --end="$END"

How it can fail

Five failure modes specific to cluster-aware inhibition:

1. Missing cluster in equal:. Symptom: a staging host-down suppresses a production service-down because both share the alertname. Investigation: dump /api/v2/alerts?silenced=true and check the cluster label on the source of every suppression.
2. env label is not present on the source. Symptom: the staging/production separator rule does not fire because the source has no env label. The Prometheus rule that produces the alert forgot to set it. Investigation: curl /api/v2/alerts?active=true for the source alertname; confirm env is in .labels.
3. Inconsistent cluster labels across Prometheus instances. Symptom: two Prometheuses in the same cluster emit different cluster values (prod-eu vs production-eu). Suppressions do not fire because the labels never agree. Investigation: diff the external labels across every Prometheus job.
4. Global rule without a comment. Symptom: a reviewer six months later does not know whether the global blast radius is intentional. The rule is either too broad (and silently masking a regional failure) or correct (and depends on tribal knowledge). Investigation: every rule without equal: should have an explicit comment that names the exception.
5. Region rule with cluster in equal: by mistake. Symptom: a regional upstream outage affects two clusters in the region; only one cluster’s dependents are suppressed. The other cluster’s dependents page the on-call. Investigation: confirm the equal: clause on every region-wide rule does not contain cluster.

How to troubleshoot it

The diagnostic order when a suppression crosses a boundary it should not.

1. Is the source alert from the right cluster? curl /api/v2/alerts?active=true&filter=alertname=HostDown. Inspect .labels.cluster on every HostDown alert. If a staging HostDown is the source of a production suppression, the rule is missing cluster in equal:.
2. Is the target alert in the same cluster? Compare .labels.cluster between source and target. They must agree for a single-cluster rule to fire.
3. Is the rule scoped correctly? Read the rule’s source matchers and equal clause. Confirm the env, cluster, and region filters match the intent.
4. Are the labels consistent across Prometheus instances? Diff the external labels across every Prometheus job. Two different cluster label conventions in the same logical cluster means the rules never fire.
5. Read the alertmanager log. Run with --log.level=debug. The lines around matched source and evaluated rule show which alert pairs were considered and which were suppressed.

Security implications

The cluster, region, and env labels are operationally sensitive. The labels expose the topology of the platform, the geographies served, and the staging/production boundary. Restrict access to the Prometheus rules repository and to the Alertmanager API accordingly.

A staging rule with no env filter is also a security concern: it means staging alerts can suppress production alerts, which is both an operational failure and a confidentiality concern (the staging alert may carry internal-only labels).

Performance implications

The cost of adding cluster and region to the equal clause is constant per evaluation. The cost of maintaining the labels is non-trivial: every Prometheus instance must agree on the label vocabulary, and every new cluster requires a corresponding entry in the configuration. Treat the label vocabulary as production configuration that is version- controlled alongside the alert rules.

Production guidance

• Add cluster to the equal clause of every rule by default. Remove it only after explicit review.
• Add env to the source and target matchers of every staging/production rule. The separator is mandatory.
• Use region for rules that legitimately cross clusters within a region. Do not use it as a substitute for cluster on single-cluster rules.
• Require an explicit comment on every rule without equal:. The comment names the exception and the review date.
• Validate the staging/production separator end-to-end after every change. The dry-run above takes 30 seconds and catches the most expensive mistake.

Verification

You should now be able to answer:

• Why is cluster in the equal: clause the default for single-cluster rules?
• What is the staging/production separator, and why is it mandatory?
• When is a region-wide rule the right pattern, and what is the equal clause?
• How do you prove the staging/production separator holds with a live dry-run?

Quiz