Who Monitors the Monitoring

A primary database crashes at 02:47. The application error rate triples. The on-call engineer is never paged. The reason is not that the alert did not fire; the alert fired, went into Alertmanager, hit a misconfigured webhook, and never reached the paging integration. The dashboard stayed green because nobody had ever built a metric that asked “did the pager actually receive this alert?”. The platform was monitoring the application but not itself.

This is the meta-monitoring problem. The platform that observes production must itself be observed, by something that does not depend on it.

What it is

Meta-monitoring is the discipline of observing the observability stack. It answers the question “is the monitoring platform working?” using a chain that does not depend on the platform being monitored.

Three properties distinguish a meta-monitoring setup from a regular one:

1. Independence. The observer cannot share a failure mode with the observed. A Prometheus that scrapes another Prometheus on the same host is not meta-monitoring; it is co-monitoring.
2. Out-of-band delivery. The page about a broken Prometheus must reach the operator through a path that does not depend on Prometheus. Email via a third-party relay, an SMS provider with its own status page, a hardware watchdog, or a public uptime probe.
3. Ownership. A name attached to the meta-monitoring setup. “Whoever is on call for the platform” is not a name. A team or a named individual who knows the dashboards, the alerts, and the runbook.

The most common shape in a mid-sized organisation (10-100 services, 1-3 platform engineers) is one of:

• Single-prometheus-with-watchdog. One Prometheus server. A small external uptime probe (Dead Man’s Snitch, Healthchecks.io, Better Stack, or a self-hosted equivalent) pings the Prometheus alert endpoint on a tight schedule. If the ping stops, the probe alerts the on-call. The probe is the only thing in the chain outside the cluster.
• Two-prometheus cross-scrape. Two independent Prometheus servers, each scraping the other and Alertmanager. Each runs its own rules. Each is configured to page if the other goes silent. Both pages route through a single Alertmanager cluster but the watchdog alert (no alerts received for N minutes) is delivered out-of-band.
• Full meta-stack. Prometheus plus Loki plus Tempo plus Grafana, each scraped by a separate “observer” Prometheus on a separate host in a separate failure domain, with a shared watchdog probe.

The first shape is the minimum. The second is what most teams grow into within six months of going to production. The third is what a regulated environment (finance, healthcare, public sector) runs.

Why a sysadmin cares

Every incident response in a modern shop starts with the monitoring platform. If the platform is silent or wrong, the incident lasts longer, the wrong team is paged, or nobody is paged at all. The recursive failure (“the thing that watches the thing is broken”) is one of the most expensive classes of outage because it is invisible by construction: the dashboards are green precisely because the dashboards are broken.

The single most common shape of this failure is a paging integration that stops working. The application alerts fire, hit Alertmanager, but the webhook URL is wrong or the SMTP relay is down. The application team does not find out until a user reports the problem. Meta-monitoring catches this within minutes instead of within hours.

How it works

The recursive problem has three layers:

+----------------------------------------------+
| Out-of-band delivery (SMS, third-party ping) |
+--------------------+-------------------------+
                     |
                     v
+----------------------------------------------+
| Watchdog probe (Dead Man's Snitch,           |
| Healthchecks.io, external uptime check)      |
+--------------------+-------------------------+
                     |
                     v
+----------------------------------------------+
| Observer Prometheus (separate host / account)|
| Scrape jobs:                                   |
|   - primary-prometheus /metrics                |
|   - alertmanager /metrics                      |
|   - loki /metrics                              |
|   - tempo /metrics                             |
|   - grafana /metrics                           |
|   - node-exporter on each platform host        |
| Alerting rules:                                |
|   - up == 0                                    |
|   - rule_evaluation_failures_total             |
|   - watchdog (no alerts received for 10m)      |
+--------------------+-------------------------+
                     |
                     v
+----------------------------------------------+
| Primary platform (the thing being observed)  |
+----------------------------------------------+

The watchdog probe is the smallest piece and the most important. It is a single ping from outside the cluster that proves “the alerting chain is alive end to end”. The probe does not need to know what the alerts say; it only needs to know that some alert was emitted inside the time window. If the probe stops receiving pings, the operator is paged out-of-band.

The two-Prometheus cross-scrape

The two-Prometheus pattern is a specific implementation:

+-------------+        scrape         +-------------+
| prom-A      | <--------------------- | prom-B      |
| (primary)   | ---------------------> | (observer)  |
+-------------+   scrape              +-------------+
       |                                    |
       v                                    v
+-------------+                        +-------------+
| Alertmgr-A  | -- gossip -->          | Alertmgr-B  |
+-------------+                        +-------------+

Each Prometheus runs its own scrape jobs and its own rule files. Each Prometheus has its own alertmanager cluster member. The two Alertmanager clusters gossip over their cluster.peer positions. The two Prometheus servers can also be HA replicas (see Part VI), but the meta-monitoring role requires that one of them is configured to alert on the silence of the other.

In practice, one of the two is the “observer”. The observer’s rules include the four primary metrics:

• up == 0 on every scrape job it knows about
• rate(prometheus_rule_evaluation_failures_total[5m]) greater than zero
• absent(up{job="alertmanager"}) for at least one minute
• prometheus_config_last_reload_success_timestamp_seconds earlier than ten minutes ago

The primary’s rules do not need to include meta-monitoring. The observer’s job is to monitor the primary.

Under the hood

How to configure it

Observer Prometheus scrape config

# /etc/observer/prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    observer: prom-b
    cluster: meta

rule_files:
  - /etc/observer/rules/*.yml

scrape_configs:
  # The primary Prometheus, on its own host.
  - job_name: primary-prometheus
    static_configs:
      - targets:
          - prom-primary.internal:9090
        labels:
          component: prometheus
          role: primary

  # Alertmanager, on its own host. AM exposes /metrics on 9093.
  - job_name: alertmanager
    static_configs:
      - targets:
          - alertmanager-1.internal:9093
          - alertmanager-2.internal:9093
        labels:
          component: alertmanager

  # Loki, Tempo, Grafana - each gets a job.
  - job_name: loki
    static_configs:
      - targets: [loki-1.internal:3100]
        labels: { component: loki }
  - job_name: tempo
    static_configs:
      - targets: [tempo-1.internal:3200]
        labels: { component: tempo }
  - job_name: grafana
    metrics_path: /metrics
    static_configs:
      - targets: [grafana-1.internal:3000]
        labels: { component: grafana }

  # node_exporter on each platform host. Disk and CPU are part
  # of the platform health, not just the platform service.
  - job_name: node
    static_configs:
      - targets:
          - prom-primary.internal:9100
          - loki-1.internal:9100
          - tempo-1.internal:9100

Observer alerting rules

# /etc/observer/rules/meta.yml
groups:
  - name: meta-prometheus
    rules:
      # 1. Primary Prometheus is dead.
      - alert: PrimaryPrometheusDown
        expr: up{job="primary-prometheus"} == 0
        for: 2m
        labels: { severity: critical, team: platform }
        annotations:
          summary: "Primary Prometheus unreachable from observer"
          description: |
            Observer Prometheus cannot reach the primary
            Prometheus at {{ $labels.instance }}. Application
            alerts may be silently failing. Investigate
            immediately.

      # 2. Primary Prometheus cannot evaluate rules.
      - alert: PrimaryPrometheusRuleEvalFailing
        expr: |
          rate(prometheus_rule_evaluation_failures_total[5m]) > 0
        for: 5m
        labels: { severity: critical, team: platform }

      # 3. Primary Prometheus configuration did not reload.
      - alert: PrimaryPrometheusConfigReloadFailed
        expr: |
          prometheus_config_last_reload_success_timestamp_seconds
            < (time() - 600)
        for: 10m
        labels: { severity: warning, team: platform }

      # 4. Alertmanager cluster is split or down.
      - alert: AlertmanagerClusterUnhealthy
        expr: |
          alertmanager_cluster_health_score{job="alertmanager"}
            < 0.5
        for: 2m
        labels: { severity: critical, team: platform }

Watchdog probe

The watchdog is the third leg. It does not need to be Prometheus-based; it only needs to fire when no meta-alert has fired in a long time. The canonical implementation is a Prometheus alert that pages everywhere on a heartbeat, and an external uptime probe that pages if the heartbeat stops.

# A heartbeat alert that fires every minute, on purpose.
- alert: ObserverHeartbeat
  expr: vector(1)
  for: 1m
  labels:
    severity: info
    heartbeat: "true"
  annotations:
    summary: "Observer Prometheus is alive"

The watchdog probe (Dead Man’s Snitch, Healthchecks.io, Better Stack, or similar) is configured to expect a ping every minute. If the ping stops for three minutes, the probe pages via SMS. The ping is delivered by an alertmanager receiver that forwards to the probe URL.

How to validate it

Confirm the observer Prometheus is actually scraping the primary:

# READ-ONLY
curl -s http://observer.internal:9090/api/v1/targets \
  | jq '.data.activeTargets[] | {job: .labels.job, health: .health}'

A healthy output shows health: up for every job, including primary-prometheus and alertmanager. The lastScrape field should be within the last 30 seconds.

Confirm the rules are loaded and evaluating:

# READ-ONLY
curl -s http://observer.internal:9090/api/v1/rules \
  | jq '.data.groups[].rules[] | {name: .name, state: .lastEvaluation, health: .health}'

Each rule shows a recent lastEvaluation and a health of ok or err. A rule that has health: err is failing to evaluate and should be investigated before the alert is needed.

Confirm the watchdog chain end-to-end:

# READ-ONLY - verify the probe received a ping in the last 2m.
curl -s https://healthchecks.io/api/v3/checks/ \
  -H "X-Api-Key: $HC_TOKEN" \
  | jq '.checks[] | {name: .name, status: .status, last_pinged: .last_ping_at}'

status: up and a recent last_pinged timestamp prove the chain is alive. A status: down or stale last_pinged means something in the chain broke and the watchdog has already paged.

How it can fail

Six shapes appear repeatedly in production meta-monitoring.

1. The observer shares a failure domain with the primary

The observer runs on the same VM, the same Kubernetes node, the same availability zone, or the same network switch as the primary. When the zone partitions, both go silent together. The watchdog never fires because there is no second observer. Symptom: the cluster appears healthy to itself until the zone comes back and the alerts from 02:00 replay in a single burst.

2. The watchdog page goes through the same integration

The Dead Man’s Snitch (or equivalent) page is configured to deliver via the same Slack channel or the same on-call rotation as the application alerts. When Slack has an outage, both fail together. Symptom: application alerts stop, but the operator does not find out until a user does. The fix is to deliver the watchdog via SMS to a phone that is not in the same paging provider.

3. The meta-monitoring rules are never tested

The observer Prometheus is set up once and never re-tested. A rule that referenced a renamed metric silently stops firing. Symptom: months pass without a single meta-alert; the team believes the chain is healthy because nobody has checked.

4. The “who” is unclear

The platform team’s runbook says “the on-call engineer is responsible for meta-monitoring.” Nobody is named. Symptom: when the page fires, three engineers open the runbook at once and none of them recognise the failure because the last person to debug it left the company. The fix is to attach the meta-monitoring setup to a specific team and a specific rotation, the same way application alerts are owned.

5. The observer and the primary are HA replicas of each other

The two-Prometheus pattern is implemented as a Thanos or Mimir-style HA pair. Both replicas are configured identically. When the underlying storage is the issue, both fail to evaluate. Symptom: no meta-alert fires; the storage failure is only discovered when an application alert is missed. The fix is to make the observer a separate server with separate storage and separate rules, not a HA replica of the primary.

6. The observer is the only thing monitoring the observer

Recursion again. The observer Prometheus is configured to alert on its own health, but nothing outside the cluster confirms it is alive. Symptom: the observer can be down for hours without anybody knowing, because no other system is checking. The fix is the third leg of the stool: the watchdog probe, delivered out-of-band.

How to troubleshoot it

Security implications

The observer Prometheus holds credentials to scrape the primary platform’s /metrics endpoints. If the primary is mTLS-protected or basic-auth-protected, the observer needs the matching client certificate or password. Rotate these the same way application secrets are rotated.

The watchdog probe URL is a secret. It is a credential that, if leaked, can be used to suppress alerts by injecting pings without the alertmanager chain being alive. Store the probe URL in a secret manager, not in plaintext in a config repo.

The meta-monitoring alerts page humans. The page itself can be spoofed. If the watchdog probe supports HMAC signing of pings, enable it. If the page is delivered via a webhook, sign the webhook with the receiver’s HMAC key.

Performance implications

Meta-monitoring is cheap if it is scoped correctly. The observer Prometheus scrapes a small, fixed set of targets (the platform services, not every application service). Series count is in the low thousands, not the millions. Local TSDB retention can be 24-72 hours; long retention is not needed because meta-alerts look back at most 30 minutes.

The risk is scope creep. The observer starts as a meta-monitoring Prometheus and slowly becomes the second HA replica of the primary. Once that happens, the meta-monitoring role is gone and the operator has a worse version of HA. Keep the observer’s rule files separate, the scrape jobs separate, and the storage separate.

Production guidance

• Pick a watchdog probe that lives outside your primary cloud provider.
• Deliver the watchdog page via a different channel than application alerts (SMS via a different provider, hardware pager, phone tree).
• Name a team and a rotation. The platform team usually owns meta-monitoring. Document the runbook.
• Re-test the meta-alerts quarterly. Drop a known-bad rule on the primary and confirm the observer fires.
• Keep the observer’s rule files in their own git repo or directory. They are not application rules.
• Treat the observer as a separate failure domain. Different host, different zone, different provider if possible.

Verification

You should now be able to answer:

• What is the operational difference between meta-monitoring and regular monitoring?
• What are the three required properties of a meta-monitoring setup (independence, out-of-band delivery, ownership)?
• What is the watchdog probe and why does it live outside the cluster?
• Name two failure shapes that meta-monitoring catches and regular monitoring misses.
• What is the most common implementation of meta-monitoring in a mid-sized production environment?

Quiz