Meta Alerts

A page fires at 04:00: PrometheusTargetDown for a critical application. The on-call engineer opens Grafana. The dashboard is green; the application is fine. The engineer dismisses the page as a false positive. Five minutes later the actual page arrives: CheckoutServiceDown. The application is genuinely down. The on-call has already burnt the first five minutes of the mitigation window chasing the wrong thing.

The first page was a meta alert. It came from the meta-Prometheus because the production Prometheus had stopped scraping one of its targets. The production Prometheus itself did not fire the page; the meta did, because the meta’s job is to notice when the production stack is degraded in a way the production stack cannot self-report. The on-call dismissed it because the application dashboard was green, which is the symptom the production Prometheus was supposed to detect — but the production Prometheus was the broken thing.

This lesson is about the discipline of meta alerts: what to alert on, how to route them, and how to keep the meta alert path from sharing fate with the production alert path.

What it is

A meta alert is an alert generated by the meta-Prometheus about the state of the production monitoring stack. The subject is a platform component — Prometheus itself, Alertmanager, the OTel Collector, a Loki ingester, a Tempo distributor — not an application. The routing path is the meta-Alertmanager cluster, not the production Alertmanager cluster. The notification destination may be the same on-call rotation (often is) but the path that gets the page there is independent.

The distinction matters because the production alert path is what the meta alert is about. A meta alert that routes through the production Alertmanager is a meta alert that has signed up for the same failure modes as the production alerts. A meta alert that routes through a separate meta-Alertmanager is a meta alert that can detect when the production alerting path is broken.

Why a sysadmin cares

Meta alerts are the only mechanism that can detect a class of failures the production stack cannot self-report. There are four such classes in any production observability stack:

1. Prometheus is down. No alerts fire, because the rule evaluator is the thing that is broken. The meta, watching Prometheus from outside, can detect up{job="prometheus"} == 0 and page.
2. Scrape failures on a target. Prometheus emits up == 0 and records scrape errors, but if the alert rule itself is wrong (typo, label mismatch), no page fires. The meta, watching the scrape failure rate, can detect a sustained scrape failure pattern even when the production rule is broken.
3. Rule evaluation stalls. A high-cardinality recording rule or a runaway join can stall the rule evaluator. No alerts fire, because evaluation is the broken thing. The meta, watching prometheus_rule_evaluation_duration_seconds, can detect the stall before it becomes a multi-hour outage.
4. Alertmanager is degraded. Alertmanager’s gossip cluster can lose quorum; the notifier can fall behind. The production Prometheus still evaluates rules and pushes alerts, but the pages do not arrive. The meta, watching the Alertmanager cluster, can detect the degraded state and page through a different path.

Without meta alerts, each of these is a silent failure. With meta alerts, each is a paged failure with a five-minute time-to-detect.

How it works

The discipline has four parts: the alert subject, the alert expression, the alert route, and the alert test.

   Subject       Expression               Route          Test
   -----------   ----------------------   ------------   -------------
   Prometheus    up{job="prometheus"}     meta-AM-1,     stop prod-prom,
   itself        == 0 for 5m              meta-AM-2      confirm page
                                                       arrives in <5m

   Scrape        rate(scrape_series_added  meta-AM-1,     block network
   failures      [5m]) == 0 for 10m       meta-AM-2      between prod-prom
                 on a critical job                       and target, confirm
                                                          page arrives

   Rule          prometheus_rule_evalu    meta-AM-1,     deploy runaway
   evaluation    ation_duration_seconds   meta-AM-2      rule on prod,
   stall         quantile 0.99 > 10s                       confirm page
                                                          arrives

   Alertmanager  ALERMA_                   meta-AM-1,     kill one AM pod,
   degraded      {alertmanager_           meta-AM-2      confirm page
                 cluster_health}                            arrives via
                                                          gossip failover
                 or alertmanager_
                 notifier_errors_total
                 rate > 0

The route always terminates at a meta Alertmanager that is independent of the production Alertmanager. The notification destination is the same on-call rotation; the channel into the rotation is the part that is independent.

How to configure it

The meta alert rules live in the meta-Prometheus rule_files:. The alert routes live in the meta-Alertmanager alertmanager.yml. Both must be on the management infrastructure, not the production infrastructure.

Meta alert rules

# /etc/meta-prometheus/rules/meta-alerts.yml
groups:
- name: meta-alerts
  interval: 30s
  rules:

  - alert: MetaPrometheusDown
    expr: |
      sum by (instance, prometheus_cluster) (
        up{job="prometheus",cluster="meta"} == 0
      ) > 0
    for: 2m
    labels:
      severity: page
      team: platform
      layer: meta
    annotations:
      summary: 'Production Prometheus unreachable from meta'
      description: |
        The meta-Prometheus has been unable to scrape
        {{ $labels.instance }} ({{ $labels.prometheus_cluster }})
        for 2 minutes. The production Prometheus is either
        down or unreachable.
      runbook_url: 'https://runbooks.example.com/meta/prom-down'
      dashboard_url: 'https://grafana-meta.example.com/d/meta'

  - alert: MetaPrometheusScrapeFailure
    expr: |
      sum by (job, instance) (
        rate(prometheus_target_scrapes_exceeded_body_size_limit_total[5m])
      ) > 0
    for: 10m
    labels:
      severity: ticket
      team: platform
      layer: meta
    annotations:
      summary: 'Production Prometheus scrape failure rate elevated'

  - alert: MetaPrometheusRuleEvalStall
    expr: |
      histogram_quantile(0.99,
        sum by (le) (
          rate(prometheus_rule_evaluation_duration_seconds_bucket[5m])
        )
      ) > 10
    for: 10m
    labels:
      severity: page
      team: platform
      layer: meta
    annotations:
      summary: 'Rule evaluation p99 above 10s for 10m'

  - alert: MetaAlertmanagerDegraded
    expr: |
      sum by (instance) (
        up{job="alertmanager",cluster="meta"} == 0
      ) > 0
    for: 2m
    labels:
      severity: page
      team: platform
      layer: meta
    annotations:
      summary: 'Meta Alertmanager {{ $labels.instance }} unreachable'

  - alert: MetaPrometheusDiskPressure
    expr: |
      prometheus_tsdb_head_series
      / on (instance) (
        prometheus_target_sync_failed_total
      )
      > 100000
    for: 30m
    labels:
      severity: ticket
      team: platform
      layer: meta
    annotations:
      summary: 'Production TSDB head block above 100k series for 30m'

Meta Alertmanager routing

# /etc/meta-alertmanager/alertmanager.yml
route:
  receiver: meta-platform-pager
  group_by: [alertname, instance]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - severity = "page"
        - layer = "meta"
      receiver: meta-platform-pager
      continue: false
    - matchers:
        - severity = "ticket"
        - layer = "meta"
      receiver: meta-platform-ticket
      continue: false

receivers:
  - name: meta-platform-pager
    pagerduty_configs:
      - routing_key:
          from_file: /etc/meta-alertmanager/secrets/pager.key
        severity: 'critical'
        description: '{{ .CommonLabels.alertname }} on {{ .CommonLabels.instance }}'

  - name: meta-platform-ticket
    slack_configs:
      - api_url:
          from_file: /etc/meta-alertmanager/secrets/slack.url
        channel: '#platform-meta'
        title: '{{ .CommonLabels.alertname }}'
        text: '{{ .CommonAnnotations.description }}'

Reading the configs:

• for: on each rule is longer than for application alerts. Meta alerts about the production stack firing every minute is itself a noise source; ten minutes for an evaluation stall is reasonable because the stall is a slow-burn problem.
• team: platform, layer: meta — the owner label and a label that explicitly distinguishes meta alerts from production alerts in the routing tree.
• pagerduty_configs.routing_key from a file, not inline. PagerDuty routing keys are secrets and rotate.
• severity: page and severity: ticket routes are explicit. A meta alert that pages the on-call rotation should not silently re-classify as a ticket because a parent route matched first.

How to validate it

Five checks confirm the meta alerts are wired correctly.

# SEVERITY: READ-ONLY
# 1. Validate the meta alert rules syntax.
promtool check rules /etc/meta-prometheus/rules/meta-alerts.yml

Expected output:

SUCCESS: /etc/meta-prometheus/rules/meta-alerts.yml
        4 rules found

# SEVERITY: READ-ONLY
# 2. Confirm the meta-Alertmanager is on a separate cluster
#    from the production Alertmanager. The peer list should
#    contain only meta-Alertmanager instances.
curl -s http://meta-alertmanager:9093/api/v1/status \
  | jq '.data.cluster.status'

Expected output: ready. The cluster name should be meta-alertmanager (or whatever you set in --cluster.name); not the production cluster name.

# SEVERITY: READ-ONLY
# 3. Confirm the routes exist in the meta-Alertmanager.
curl -s http://meta-alertmanager:9093/api/v1/status \
  | jq '.data.config.original' | grep -A2 'meta-platform-pager'

The output should show the route block from the YAML.

# SEVERITY: SERVICE-IMPACT (controlled test on a non-prod meta)
# 4. Confirm a meta alert actually fires when the production
#    Prometheus is unreachable. From a host that can reach the
#    meta but not the production Prometheus, simulate the
#    failure by stopping the production Prometheus for 5 minutes.
#    Confirm the meta alert fires in the meta-Alertmanager.
curl -s http://meta-alertmanager:9093/api/v2/alerts \
  | jq '.[] | select(.labels.alertname=="MetaPrometheusDown")'

A non-empty result confirms the end-to-end path: meta scrape, rule evaluation, alert dispatch, Alertmanager receipt.

# SEVERITY: READ-ONLY
# 5. Confirm the page arrives at the notification destination.
#    PagerDuty / Opsgenie / Slack API can be queried for
#    recent incidents on the meta service.
pd-cli incidents list --service meta-platform --since 10m

The list should include the MetaPrometheusDown incident from step 4.

How it can fail

Six failure modes recur.

1. Meta alerts route through the production Alertmanager. Symptom: the production Alertmanager is down, the meta alert about the production Alertmanager is in flight, and the page is queued in a cluster that is also down. The page does not arrive.
2. Meta alert has no for: clause. Symptom: the alert fires on every scrape interval, the Alertmanager receives duplicate alerts, the on-call is paged repeatedly for a condition that self-resolves.
3. Meta alert has no runbook URL. Symptom: the page arrives with a useful summary but no actionable context. The on-call acknowledges and waits for more information, burning the mitigation window.
4. Meta alert has no owner label. Symptom: Alertmanager falls through to the default catch-all route, the page arrives at a generic on-call rotation that does not own the platform, and the right team learns about the incident from a third party.
5. Meta alert subject is the application, not the platform. Symptom: the meta fires CheckoutServiceDown based on a meta-Prometheus query against production metrics. The meta is acting as a parallel Prometheus, not as a meta-monitoring platform. The alert is correct but the discipline is wrong; the meta is now a second source of truth for application state.
6. Meta alert is silenced indefinitely. Symptom: the alert has been silenced for 90 days because it kept firing during a known-bad period. The silencer has left the team. The alert is now a long-lived silence. The right disposition is to redesign the alert or delete it; silencing without a fix is a deferred deletion.

How to troubleshoot it

When a meta alert misbehaves, work the path from the meta outward.

1. Confirm the rule is loaded and inactive. promtool check rules for syntax; curl /api/v1/rules for state. If the rule is missing, the file is not in the meta-Prometheus rule_files: glob.
2. Confirm the rule evaluates. Run the expr against the meta’s query API. If the expression returns no series, the metric is not being federated or scraped correctly.
3. Confirm the alert state. curl /api/v1/alerts on the meta. If state is inactive, the rule is loaded and evaluating but the condition does not hold. If state is pending, the for: duration has not elapsed. If state is firing, the alert has been pushed to the meta-Alertmanager.
4. Confirm the alert arrived in the meta-Alertmanager. curl http://meta-alertmanager:9093/api/v2/alerts. If the firing alert is not present, the meta-Alertmanager push failed; check the meta-Prometheus alerting: config and the network path.
5. Confirm the route matched. Inspect the meta-Alertmanager UI for the alert’s label set and the route tree. If no route matched, the receiver is null and the page is dropped.
6. Confirm the notification destination. Query PagerDuty / Opsgenie / Slack for recent incidents. If no incident appears, the receiver config is wrong; check the routing_key or webhook URL.

Security implications

The meta-Alertmanager has access to the notification destination credentials: PagerDuty routing keys, Slack webhook URLs, Opsgenie API tokens. A compromise of the meta-Alertmanager is a compromise of the on-call notification path. The meta-Alertmanager should be hardened as if it were production:

• The notification secrets are read from files, not embedded in the YAML.
• The Alertmanager API is restricted to the management network. amtool access requires an API token, and the token rotates.
• Silences are auditable. Every silence has an author, a comment, and an expiry. A silence with no comment is a process smell.

Performance implications

The meta-Prometheus is small; the meta-Alertmanager is smaller. The meta alert rules are evaluated on a longer interval than the application rules — thirty seconds is typical — because the subjects of meta alerts (production platform state) change slowly. The meta-Alertmanager holds the alerts from the meta-Prometheus plus the silences applied to them. Silences expire automatically; unmanaged silences are a memory leak, not a feature.

Production guidance

• The meta-Alertmanager is a separate cluster with at least two replicas behind a DNS name. A single replica is a single point of failure for the meta alerting path.
• Meta alert for: durations are longer than for application alerts. Meta alerts about a slowly-degrading platform do not need to fire on a thirty-second scrape.
• Meta alert owners are explicit. team: platform, layer: meta, and a runbook_url annotation that points to a runbook the platform team owns.
• Meta alert routing uses the same notification destination as production alerts (the same PagerDuty service, the same Slack channel) but a different cluster. The destination is the only shared component.
• Silences on meta alerts have the same lifetime discipline as silences on production alerts. A silence older than 30 days is a deferred deletion.

Verification

You should now be able to answer:

• What is a meta alert, and how does its subject differ from a production alert?
• What are the four canonical meta alerts any production stack must have?
• Why must meta alerts route through a meta-Alertmanager cluster that is separate from the production Alertmanager cluster?
• What is the right test to confirm a meta alert is wired correctly end-to-end?
• What is the first thing to check when a meta alert fires but the page does not arrive?

Quiz