Meta-Monitoring Basics

A page fires at 03:00: PrometheusTargetDown{job="checkout", instance="checkout-7"}. The on-call engineer opens Grafana to investigate. Grafana renders the dashboard panels from Prometheus, but every panel is grey. The query tool returns no data. The alert came from Prometheus, so Prometheus must be working — but it is not the Prometheus the engineer is looking at. It is a separate Prometheus, the meta-Prometheus, that has been watching the production Prometheus for six months and is now reporting that the production Prometheus cannot scrape its targets. The Grafana the engineer opened is pointed at the production Prometheus, which is silent. The dashboard the engineer is reading is wallpaper.

This is the most common shape of a meta-monitoring incident: the meta is the only thing that knew, and the meta told someone, but the operator looked at the wrong place. The discipline exists to keep that gap closed.

What it is

Meta-monitoring is the discipline of watching the monitoring platform itself. The platform is everything that produces, stores, routes or alerts on telemetry: Prometheus servers, Loki ingesters, Tempo distributors, the OpenTelemetry Collector, Grafana Alloy, Grafana, Alertmanager, and the storage backends they depend on. Meta-monitoring answers questions about those components from their own telemetry, not from application telemetry.

The distinction matters because the failure modes are different. Application monitoring asks “is the service behaving?” Meta- monitoring asks “is the watcher behaving?” When the watcher is broken, every question it would have answered is unanswerable — including the question “is the watcher broken?”

Why a sysadmin cares

The cost of not having meta-monitoring is asymmetric. The production Prometheus can be down for hours before anyone notices, because the dashboards that would have shown the outage are served by the same Prometheus. The on-call engineer investigates through a panel that is rendering an empty result set and concludes “nothing is wrong, the page was a false positive.” The real outage continues. The actual incident is the meta-monitoring gap, not the scrape failure the meta would have caught.

The four operational consequences:

1. Silent alerting. Rules do not evaluate. Pages stop arriving. The team assumes the platform is healthy.
2. Silent history. New data is not written; old data is not queryable. The post-mortem cannot answer “when did this start?” because the answer is in a dead ingester WAL.
3. Silent dashboards. Panels render the last sample held in cache, then go grey. The operator assumes the service is stable because the panel is calm.
4. Misattribution. When the platform is intermittently failing, applications look intermittently broken. Engineers investigate the application, find nothing, and burn time on a non-existent bug.

None of these are visible to the application layer. All of them are visible to the platform layer. That visibility is the discipline.

How it works

The mental model is a second Prometheus watching the first. The second Prometheus is independent of the first: different hosts, different storage, different network, different alerting path. The production Prometheus exposes its own state on the standard /metrics endpoint, exactly like every other scrape target. The meta-Prometheus scrapes that endpoint and stores what it sees. Operators query the meta-Prometheus for “what is the production Prometheus doing right now?” and the meta answers.

   Production stack                      Meta stack
   ==============                       ==========

   App A --scrape--+
   App B --scrape--+--> Production Prometheus --/metrics--+
   App C --scrape--+       |                            |
                            +--evaluate--> rules         |
                            +--/federate--> (sometimes)  v
                                                     Meta Prometheus
                                                         |
                                                         +--evaluate--> rules
                                                         +--alert-----> Alertmanager (meta)
                                                         +--query-----> Grafana (meta view)
                                                         +--scrape---> production Alertmanager
                                                         +--scrape---> production Grafana
                                                         +--scrape---> production OTel Collector
                                                         +--scrape---> production Loki/Tempo

Three properties of the meta-Prometheus make this work:

• It watches the watcher. The set of scrape targets is the set of platform components: Prometheus, Alertmanager, Grafana, the OTel Collector, Loki distributors / ingesters, Tempo distributors. It does not scrape applications directly; scraping the same apps as the production Prometheus turns the meta into a parallel silo.
• It runs on different infrastructure. Different hosts, ideally different network segment. The point is to remove the shared-fate failure modes a co-located meta would inherit.
• It has its own alerting path. Meta alerts go through a meta-Alertmanager, on a different notification channel, to a different rotation. A production Alertmanager outage does not silence meta alerts.

How to configure it

The minimum configuration is one scrape job in the meta-Prometheus prometheus.yml that targets the production Prometheus self-metrics endpoint. The example below scrapes a single production Prometheus over HTTPS with basic auth.

# /etc/meta-prometheus/prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    cluster: meta
    replica: meta-prom-1

scrape_configs:
  - job_name: prometheus
    metrics_path: /metrics
    scheme: https
    basic_auth:
      username: meta-scraper
      # Loaded from file, not the config. See security section.
      password_file: /etc/meta-prometheus/secrets/prod-prom.pass
    static_configs:
      - targets:
          - prod-prom-1.internal:9090
          - prod-prom-2.internal:9090
        labels:
          prometheus_cluster: production
          site: dc1

  - job_name: alertmanager
    metrics_path: /metrics
    static_configs:
      - targets: ['prod-alertmanager-1.internal:9093',
                  'prod-alertmanager-2.internal:9093']
        labels: { component: alertmanager }

  - job_name: grafana
    metrics_path: /metrics
    static_configs:
      - targets: ['prod-grafana-1.internal:3000']
        labels: { component: grafana }

Reading the config:

• external_labels.cluster: meta — every series the meta-Prometheus produces is labelled with cluster="meta". When you query the meta, you can tell at a glance that the data is meta-monitoring data, not production telemetry.
• prometheus_cluster: production — set as a static_config label so queries can join on prometheus_cluster to identify which production instance is being watched.
• password_file: not password: — the credential is read from a file with restricted permissions (0640, owned by the Prometheus user). Inline credentials in the YAML are a security smell and break every secret scanner.
• Two production targets in prometheus — scraping both production replicas independently lets the meta detect a scrape-path failure on one of them.

How to validate it

Four checks confirm the meta is alive and pulling what you expect.

# SEVERITY: READ-ONLY
# 1. Validate the configuration syntax.
promtool check config /etc/meta-prometheus/prometheus.yml

Expected output:

SUCCESS: /etc/meta-prometheus/prometheus.yml

# SEVERITY: READ-ONLY
# 2. Confirm the meta is scraping the production Prometheus.
#    up == 1 means the meta reached the target and got a 200 OK.
curl -s http://meta-prom:9090/api/v1/query \
  --data-urlencode 'query=up{job="prometheus",cluster="meta"}' \
  | jq '.data.result[] | {instance: .metric.instance, up: .value[1]}'

Expected output:

{ "instance": "prod-prom-1.internal:9090", "up": "1" }
{ "instance": "prod-prom-2.internal:9090", "up": "1" }

# SEVERITY: READ-ONLY
# 3. Confirm the meta is receiving self-monitoring data, not just
#    connectivity probes. prometheus_tsdb_head_series is the
#    series count in the production head block.
curl -s http://meta-prom:9090/api/v1/query \
  --data-urlencode \
  'query=prometheus_tsdb_head_series{prometheus_cluster="production"}' \
  | jq '.data.result[] | .value[1]'

A value above zero, and tracking the production Prometheus series count you expect, confirms the meta is observing the right target.

# SEVERITY: READ-ONLY
# 4. Confirm the meta is querying itself for self-monitoring.
curl -s http://meta-prom:9090/api/v1/query \
  --data-urlencode 'query=up{job="prometheus",cluster="meta"}' \
  | jq '.data.result[].metric.instance' | sort -u

The list should contain the meta-Prometheus instance itself.

How it can fail

Six failure modes recur across production meta-monitoring deployments.

1. The meta-Prometheus is not running. The simplest failure. Symptom: no scrape data in the meta; the meta’s own /metrics endpoint returns connection refused; nothing in the meta dashboard updates after the meta was last restarted.
2. The meta scrapes itself but not the production stack. Symptom: up{job="prometheus",cluster="meta"} == 1 but up{job="prometheus",cluster="production"} series do not exist. The configuration was deployed with the production job commented out, or the static_config target is wrong.
3. The production Prometheus /metrics is bound to localhost. Symptom: up{job="prometheus"} == 0 from the meta’s view; production dashboards work fine from the host itself; the meta logs show scrape errors with “connection refused”. The production --web.listen-address is 127.0.0.1:9090, not 0.0.0.0:9090.
4. The scrape interval is longer than the alert latency budget. Symptom: the meta alerts fire, but they fire after the user-impacting incident has been noticed by the user. A five-minute scrape interval on a meta monitoring a five-minute SLO is useless. The interval has to be shorter than the time-to-detect budget of the worst meta-detectable failure.
5. The meta uses the same Alertmanager as production. Symptom: the production Alertmanager is down, the meta detects it, the meta alert fires into the same down Alertmanager, and nothing is paged. This is the circular dependency the next lesson handles.
6. The meta is a parallel silo. Symptom: the meta scrapes the same applications as the production Prometheus, has its own dashboards, and never scrapes the production Prometheus itself. The meta is “a second Prometheus,” not meta-monitoring.

How to troubleshoot it

When the meta-monitoring dashboard shows something unexpected, work the layers from the meta outward.

1. Confirm the meta-Prometheus is running. systemctl status meta-prometheus (or kubectl get pods -n meta). If it is not running, restart it before investigating further; everything below assumes the meta process is up.
2. Confirm the meta is querying itself. up{job="prometheus",cluster="meta"} from the meta’s own query API. If up == 0, the meta is not even healthy.
3. Confirm the meta is reaching the production Prometheus. up{job="prometheus",prometheus_cluster="production"}. If up == 0, the scrape is failing. Inspect the meta logs for the scrape error; check the target endpoint from the meta host with curl -k https://prod-prom:9090/metrics.
4. Confirm the meta is receiving the right metrics. prometheus_tsdb_head_series{prometheus_cluster="production"} should be non-zero and reasonable. If zero, the scrape succeeded but a relabel rule is dropping the data; inspect relabel_configs: in the meta’s scrape job.
5. Confirm the meta-Alertmanager is routing. curl http://meta-alertmanager:9093/-/ready should return “OK”. Inspect the meta Alertmanager log for the firing alert.
6. Form a hypothesis and find evidence. The meta logs are the first source; the meta’s own self-metrics (prometheus_*) are the second; the production Prometheus logs and metrics are the third.

Security implications

The meta-Prometheus scrapes endpoints that expose internal metrics. Those endpoints often include series with label values that name hosts, services and tenants. Treat the meta-Prometheus the same way you would treat a log aggregator with the same labels.

• Authenticate the scrape. basic_auth with a password read from a file, or mTLS. The default Prometheus /metrics endpoint has no authentication; production deployments should add --web.enable-lifecycle only after they have added an authenticating reverse proxy in front.
• Restrict the network. The meta-Prometheus should reach the production Prometheus over an internal network segment. /metrics on the public internet is a reconnaissance leak; /federate on the public internet is worse (see lesson 03).
• Restrict who can query the meta. The meta answers “is the production platform healthy?” which is information a customer of the platform would like to know. The meta’s query API and Grafana data source should be on an internal network or behind SSO.

Performance implications

The meta-Prometheus is small. It scrapes a fixed number of platform targets (Prometheus, Alertmanager, Grafana, OTel Collector, plus a handful of distributors and ingesters) on a short interval. The total series count is in the low thousands at most. The retention can be thirty days or longer for trend analysis without exceeding a few GB of disk.

The performance hazards are:

• Re-scrape of applications. If the meta is misconfigured to scrape the same applications as production, its series count jumps to the production Prometheus level and the “small” assumption breaks.
• High-cardinality platform labels. Some exporters expose labels with per-request or per-connection values. Drop them in the meta’s metric_relabel_configs before they hit the storage path.
• Long retention. A 90-day retention on the meta is fine for trend analysis; a 365-day retention is a different conversation because the meta starts to look like a full Prometheus and the cost calculus changes.

Production guidance

• Run the meta-Prometheus on different hosts from the production stack. A separate VM, a separate Kubernetes cluster, or at minimum a separate node pool. Co-location removes the point.
• Give the meta its own Alertmanager. A meta Alertmanager with two replicas behind a DNS name is the typical shape.
• Use a separate notification channel for meta alerts. The production PagerDuty service is fine; the production email distribution list is fine. The point is that a meta alert routes through infrastructure that does not depend on the production stack being up.
• Keep the meta scrape interval shorter than your worst-case time-to-detect budget. For most teams, fifteen seconds.
• Back up the meta’s TSDB. A meta-Prometheus you cannot restore is decoration.
• Document what the meta watches and what it does not. The expectation gap (“I assumed the meta caught that”) is the failure mode meta-monitoring exists to prevent, and the meta has to document its own scope to make the expectation clear.

Verification

You should now be able to answer:

• What is meta-monitoring, and how is it different from application monitoring?
• Which four operational consequences follow when meta-monitoring is missing?
• Which Prometheus endpoint exposes the production Prometheus self-monitoring metrics, and what is the minimum scrape configuration to read them?
• What is the difference between a meta-Prometheus and a parallel-silo second Prometheus?
• What is the first thing to check when the meta-monitoring dashboard shows something unexpected?

Quiz