Dependency-Aware Alerting

A production database goes down at 14:32. The database alert fires within seconds — DatabaseDown. Thirty seconds later the pager begins to receive a cascade: CheckoutServiceError, CartServiceError, PaymentServiceError, AuthServiceError. Eight alerts for one root cause. The on-call database administrator is paging about a database; every dependent service team is paging about their service.

With a dependency map, the cascade collapses. The database alert inhibits every dependent-service error alert; the on-call database engineer is the only person paged; the dependent-service teams see their alerts suppressed with a note that the cause is upstream. When the database recovers, the dependent services recover automatically.

This lesson covers the discipline of turning a service catalog into a set of labels, and the Alertmanager rules that use those labels to suppress cascades. The discipline has two design decisions: how to encode the dependency tree as labels, and how to choose between blackout (suppress entirely) and demote (reduce severity) for each dependent alert.

What it is

Dependency-aware alerting is the practice of carrying the dependency tree of a system on every alert as a set of labels, and writing Alertmanager rules that suppress or demote alerts based on whether an upstream is also alerting.

Dependency tree (from the service catalog):

  postgres-prod
      |
      +-- checkout-svc
      +-- cart-svc
      +-- payment-svc
      +-- auth-svc

Labels on every alert produced by these services:

  service="checkout-svc"
  depends_on="postgres-prod"
  depends_on_team="data-platform"
  team="checkout"

The depends_on label encodes the immediate dependency. The depends_on_team label encodes the team that owns the dependency — the team that should be paged when the dependency itself fails.

Why a sysadmin cares

The production failure shape that dependency-aware alerting prevents is the cascading incident. A single failure (a database, a load balancer, an authentication service) causes a wave of dependent failures, each of which is independently correct as an alert. The on-call rota receives N alerts for one fix. The operator must triage N alerts to find the one that matters. The mean time to identify the root cause rises with N.

A secondary failure shape is demote. Some dependent failures are not worth paging on. If the database is down, the AuthService returning 503s is not actionable — the operator cannot fix auth until the database is back. The discipline is to demote the dependent alert from warning to info for the duration of the upstream alert. The dependent alert is still recorded; it is just not paged.

How it works

There are two distinct design decisions.

Decision 1: blackout vs demote

Blackout :  suppress the dependent alert entirely
Demote   :  reduce the dependent alert severity
            (e.g. warning -> info)

Blackout is the right choice when the dependent alert carries no information beyond “the upstream is broken.” If the database is down, every ServiceErrorRateHigh on a dependent service is a direct consequence; the operator cannot act on the dependent service until the upstream is restored.

Demote is the right choice when the dependent alert carries information that is still useful during an upstream outage. For example, a LatencyHigh on a service whose upstream is slowing is informative — the operator needs to know which services are degraded even before the upstream fully fails. Demoting means “still record, do not page.”

Decision 2: how to encode the dependency

Three common patterns:

Pattern A — depends_on (label on every alert):

  service="checkout-svc"
  depends_on="postgres-prod"

  Inhibit rule:
    source_matchers: [service="postgres-prod"]
    target_matchers: [depends_on="postgres-prod"]
    equal: [cluster]

Pattern B — depends_on_team (label on every alert):

  service="checkout-svc"
  depends_on_team="data-platform"

  Inhibit rule:
    source_matchers: [team="data-platform", severity="critical"]
    target_matchers: [depends_on_team="data-platform"]
    equal: [cluster]

Pattern C — combined:

  service="checkout-svc"
  depends_on="postgres-prod"
  depends_on_team="data-platform"

Pattern A is the most precise — the rule fires only when the exact dependency is firing. Pattern B is the broadest — the rule fires for any alert owned by the dependency’s team. Pattern C is the most flexible — the precision of A and the breadth of B are both available.

Under the hood

How to configure it

A complete configuration for the four-service dependency tree above. Three rules: a per-dependency rule (Pattern A), a team-wide rule (Pattern B), and a demote rule that reduces severity rather than suppressing.

# /etc/alertmanager/alertmanager.yml
route:
  receiver: default
  group_by: [service, alertname, instance]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

inhibit_rules:
  # 1. Per-dependency blackout (Pattern A).
  #    When postgres-prod is firing critical, suppress every
  #    dependent service's error alerts on the same cluster.
  - source_matchers:
      - service="postgres-prod"
      - severity="critical"
    target_matchers:
      - depends_on="postgres-prod"
      - severity=~"warning|critical"
    equal: [cluster]

  # 2. Per-team blackout (Pattern B).
  #    When data-platform owns something critical, suppress
  #    every alert whose team depends on data-platform.
  - source_matchers:
      - team="data-platform"
      - severity="critical"
    target_matchers:
      - depends_on_team="data-platform"
      - severity="warning"
    equal: [cluster]

  # 3. Demote (Pattern A, demote variant).
  #    When the upstream is slow (warning, not critical),
  #    demote dependent latency alerts from warning to info
  #    by suppressing the warning alert entirely. The info
  #    alert (a separate recording rule with lower threshold)
  #    is allowed through.
  - source_matchers:
      - service="postgres-prod"
      - alertname=~"^(PostgresLatencyHigh|PostgresConnectionsHigh)$"
    target_matchers:
      - depends_on="postgres-prod"
      - alertname=~"^ServiceLatencyHigh$"
      - severity="warning"
    equal: [cluster]

receivers:
  - name: default
    webhook_configs:
      - url: 'http://localhost:5001/alerts'

Three things to read carefully:

1. Equal is cluster. Suppression spans every instance in the cluster. If the dependency is regional (e.g. a primary database per region), add region to the equal clause.
2. Rule 3 is a demote. The source is warning (not critical), and the target is the warning-level ServiceLatencyHigh. The warning alert is suppressed; the info-level alert (a separate recording rule) is unaffected. The dependent team still sees the latency signal in their dashboards, but is not paged.
3. severity is on both source and target. Without the target severity filter, an upstream info alert would suppress a dependent critical alert. The filter ensures the source is at least as severe as the target.

How to validate it

Five checks. The first three are static. The last two are live.

# 1. Static syntax check (CONFIGURATION severity)
amtool check-config /etc/alertmanager/alertmanager.yml

Expected output (illustrative):

Checking /etc/alertmanager/alertmanager.yml
  SUCCESS
Parsed and loaded successfully
Found inhibit rule 'postgres-down-suppresses-dependents'
Found inhibit rule 'data-platform-down-suppresses-dependents'
Found inhibit rule 'postgres-slow-demotes-dependent-latency'

# 2. SIGHUP reload (CONFIGURATION severity)
kill -HUP "$(pidof alertmanager)"

# 3. Loaded-config echo (READ-ONLY). The status API returns
#    the running configuration as a YAML string in
#    .config.original, not as a parsed object.
curl -s http://localhost:9093/api/v2/status \
  | jq -r '.config.original'

Expected output (illustrative): the three inhibit_rules entries from the file, echoed back as YAML.

# 4. Synthetic cascade dry-run (READ-ONLY)
amtool alert add alertname=PostgresDown \
  service="postgres-prod" team="data-platform" \
  cluster="prod-eu" severity="critical"

amtool alert add alertname=CheckoutServiceErrorRateHigh \
  service="checkout-svc" team="checkout" \
  depends_on="postgres-prod" depends_on_team="data-platform" \
  cluster="prod-eu" severity="warning"

amtool alert add alertname=AuthServiceErrorRateHigh \
  service="auth-svc" team="platform" \
  depends_on_team="data-platform" \
  cluster="prod-eu" severity="critical"

sleep 2
curl -s 'http://localhost:9093/api/v2/alerts?active=true&silenced=true' \
  | jq '.[] | {alertname: .labels.alertname,
                service: .labels.service,
                status: .status.state,
                inhibitedBy: .status.inhibitedBy}'

Expected output (illustrative):

{ "alertname": "PostgresDown",
  "service": "postgres-prod",
  "status": "active",
  "inhibitedBy": [] }
{ "alertname": "CheckoutServiceErrorRateHigh",
  "service": "checkout-svc",
  "status": "suppressed",
  "inhibitedBy": ["PostgresDown"] }
{ "alertname": "AuthServiceErrorRateHigh",
  "service": "auth-svc",
  "status": "suppressed",
  "inhibitedBy": ["PostgresDown"] }

Both dependents are suppressed by rule 1 (per-dependency). Rule 2 (per-team) also fires but the result is the same.

amtool has no command that deletes an alert. Alerts leave Alertmanager by resolving, not by being removed: an alert posted without an explicit end time resolves on its own after the global resolve_timeout (five minutes by default). To resolve one immediately, re-post the same label set with a start and an end that are both in the past.

# Clean up: resolve the three synthetic alerts.
START="$(date -u -d '-5 minutes' +%Y-%m-%dT%H:%M:%SZ)"
END="$(date -u -d '-1 minute' +%Y-%m-%dT%H:%M:%SZ)"
amtool alert add alertname=PostgresDown \
  service="postgres-prod" team="data-platform" \
  cluster="prod-eu" severity="critical" \
  --start="$START" --end="$END"
amtool alert add alertname=CheckoutServiceErrorRateHigh \
  service="checkout-svc" team="checkout" \
  depends_on="postgres-prod" depends_on_team="data-platform" \
  cluster="prod-eu" severity="warning" \
  --start="$START" --end="$END"
amtool alert add alertname=AuthServiceErrorRateHigh \
  service="auth-svc" team="platform" \
  depends_on_team="data-platform" \
  cluster="prod-eu" severity="critical" \
  --start="$START" --end="$END"

amtool alert query alertname=PostgresDown

# 5. Negative case — alert from a service that does NOT
#    depend on postgres (READ-ONLY)
amtool alert add alertname=PostgresDown \
  service="postgres-prod" cluster="prod-eu" \
  severity="critical"

amtool alert add alertname=SearchServiceErrorRateHigh \
  service="search-svc" depends_on="opensearch" \
  cluster="prod-eu" severity="warning"

sleep 2
curl -s 'http://localhost:9093/api/v2/alerts?active=true&silenced=true' \
  | jq '.[] | select(.labels.alertname=="SearchServiceErrorRateHigh")
         | {status: .status.state,
            inhibitedBy: .status.inhibitedBy}'

Expected output: status: "active", inhibitedBy: []. The search service does not depend on postgres and is not suppressed.

How it can fail

Five failure modes specific to dependency-aware alerting:

1. Stale dependency map. A service was migrated off Postgres six months ago, but the depends_on label still says postgres-prod. Symptom: the migrated service is silently suppressed whenever Postgres fires; the operator does not know. Investigation: compare the dependency map in the service catalog against the labels on every active alert.
2. Wrong direction. A rule says “when checkout-svc fires, suppress cart-svc” but the actual dependency is the reverse (cart-svc is upstream of checkout-svc). Symptom: during a cart outage, checkout errors are suppressed and the checkout team is not paged about the real cause. Investigation: diff every inhibit rule against the service catalog and confirm the direction of every dependency.
3. Cascade depth unaccounted for. The dependency is A -> B -> C, but the rule only handles the direct edge A -> B. Symptom: when A is down, B is suppressed but C is still paged as a downstream consequence. Investigation: trace the dependency tree from the source to the leaves and confirm every edge is covered.
4. Demote rule swallows a real signal. The demote rule suppresses the warning alert but a critical alert on the same label set is the one the operator needed. Symptom: during a slow upstream, the critical alert fires but is suppressed by a rule intended for the warning. Investigation: confirm the target severity filter is present and is at least as high as the source severity.
5. Cyclical dependency. Two services declare each other as dependencies. Each rule references the other. Symptom: every alert on either side suppresses the other; nothing reaches the pager. Investigation: walk the dependency tree and look for cycles; the service catalog should reject them at write time.

How to troubleshoot it

The diagnostic order for a missing or wrong suppression.

1. Is the source firing with the expected labels? curl /api/v2/alerts?active=true for the source. Confirm the labels the rule’s source_matchers are looking for are actually present. A source alert missing the depends_on_team label will not match a Pattern B rule.
2. Is the target firing with the expected labels? Same query, for the target. Confirm the labels the rule’s target_matchers are looking for are present. The most common cause of “the rule is not firing” is the target missing a label the author assumed was present.
3. Is the equal clause satisfiable? Diff the labels on source and target. For every label in equal:, the value must be byte-identical. The cluster label is the usual suspect (prod-eu vs prod_eu, prod vs Prod).
4. Is the dependency map current? Compare the depends_on and depends_on_team labels on every active alert against the service catalog. Any mismatch is a candidate root cause.
5. Read the alertmanager log. Run with --log.level=debug. The lines around matched source and matched target show the alert pairs that were considered and the reason each was or was not suppressed.

Security implications

The dependency map exposes the topology of the system. A service catalog that lists every internal service and every team’s dependencies is a complete map of the platform. Treat the catalog and the alert labels as production-sensitive configuration. Repository access controls apply; the Prometheus rules repo is not a public artefact.

The Alertmanager API exposes every alert label, including depends_on_team. Restrict API access to the loopback interface or a reverse proxy with authentication.

Performance implications

The cost of a dependency-aware rule is the same as a basic rule per evaluation. The new cost is the cost of maintaining the dependency map. Every service that is added, retired, or re-platformed requires a corresponding label change in every Prometheus rule that emits alerts for the service. Treat the label maintenance as part of the service’s definition-of-done: a new service is not “deployed” until its alerts carry the right depends_on and depends_on_team labels.

Production guidance

• Maintain the dependency map in one place. The service catalog is the source of truth. Every Prometheus rule imports it.
• Cover the full cascade. If A -> B -> C, every edge is a rule. The leaves still need to be reachable by an upstream alert.
• Distinguish blackout from demote explicitly. The rule’s intent — “suppress entirely” or “reduce severity” — should be obvious from its name in the configuration.
• Reject cyclical dependencies at the catalog level. An Alertmanager rule can be written for either direction; a cycle means neither rule is right.
• Review the rules when the dependency map changes. The last lesson in this module defines the rubric.

Verification

You should now be able to answer:

• What is the difference between a blackout and a demote in dependency-aware alerting?
• How does a service catalog become a set of labels on alerts?
• Why is the direction of every dependency explicit in the rule, and what happens when the direction is wrong?
• How do you prove the cascade is being suppressed end-to-end through the dependency tree?

Quiz