Proving the Meta Works

A Grafana dashboard titled “Meta-Health” rendered green for nine months. Every panel said “OK.” Every threshold was met. The on-call rotation trusted it. Then a page arrived for a production outage, and the meta-Prometheus was the thing that should have detected it. The meta-Prometheus had been down for three weeks. The dashboard said green because the dashboard’s data source was the meta-Prometheus itself. Green meant “the meta is rendering its own state,” which is not the same as “the meta is detecting production failures.” The team learned the difference during a real incident.

This lesson is about the evidence that proves the meta works. Three things are required: a synthetic test that the meta observes, a blackbox probe that exercises the production endpoints, and a Grafana dashboard that surfaces the results. The dashboard alone is not evidence; the dashboard plus the synthetic test plus the blackbox probe is evidence. Even that is not enough unless the test path bypasses the meta itself.

What it is

Meta-evidence is the set of operational signals that prove the meta-monitoring platform is doing what it was deployed to do. Three components:

1. A synthetic test target. A small service that emits a known counter on a fixed interval, scraped by both the production Prometheus and the meta. The meta should see the counter; if it stops incrementing, the meta is missing either the scrape or the storage.
2. A blackbox probe. The blackbox_exporter configured to probe the production Prometheus /metrics endpoint over HTTPS. The probe runs from a host that is not the meta host. The probe result is exposed as Prometheus metrics and scraped by a Prometheus that is not the meta.
3. A dashboard that surfaces the results. A Grafana dashboard in the management Grafana (not the production Grafana) with panels for synthetic-test status, blackbox probe status, meta-Prometheus health, and meta alert state.

The discipline is that none of the three components can be the meta itself. The dashboard reads from the meta but the dashboard rendering is not the proof. The blackbox probe exercises the production endpoints but the probe is not the proof either. The proof is a separate test path that confirms the meta observed the test.

Why a sysadmin cares

A meta-monitoring platform that has never been tested in anger is a meta-monitoring platform that does not work. The team believes it works because the dashboard is green. The dashboard is green because the meta is rendering its own state. The meta is rendering its own state because the meta process is up. None of those statements say anything about whether the meta can detect a production failure.

The cost of missing meta-evidence is the cost of discovering the gap during an incident. Three operational consequences:

1. False confidence. The dashboard is green; the team believes the platform is healthy; the actual platform is degraded; the page never arrives.
2. Slow detection. The team’s MTTD for production platform outages is the time it takes for someone to notice the dashboards are stale, which is hours. The SLO mitigation window is minutes.
3. Post-mortem surprise. The incident reveals that the meta was the silent failure mode. The fix takes weeks. The next incident is also caught by surprise.

Meta-evidence prevents all three. The cost is one synthetic test target, one blackbox probe, and one dashboard.

How it works

The mechanism is a probe-and-prove pattern. A probe runs on a host that is independent of the meta. The probe exercises the production endpoints. The probe result is stored in a place that the meta does not own. If the probe result ever shows a failure, an alert fires through a path that does not go through the meta.

   Synthetic test target            Blackbox probe host
   =====================            ===================

   meta-test-target                 meta-probe-host
        |                                |
        | emits meta_test_total          | probes production
        | every 15s                      | /metrics every 30s
        v                                v
   Production Prometheus       meta-probe-store (separate
        |                       Prometheus, separate TSDB)
        | federation             |
        v                        v
   Meta-Prometheus ----> alert ----> meta-platform-pager
        |
        | meta_test_total
        | delta == 0 for 5m
        v
   meta alert: MetaSyntheticTestStalled

The probe-and-prove pattern has four properties:

• The synthetic test target is on a separate host. Not on the meta host; not on the production stack; on a host that would survive both being down.
• The blackbox probe is from a separate host. The probe host runs the blackbox_exporter and exposes its metrics on /metrics. The probe store scrapes the probe host’s /metrics.
• The probe store is separate from the meta. A small Prometheus on the probe host, or the same probe host running Prometheus as a sidecar. The probe store is what triggers the meta-evidence alert, not the meta itself.
• The meta-evidence alert routes through a separate path. Email or SMS via a separate channel, not PagerDuty through the meta Alertmanager. The whole point is that this alert must fire when the meta cannot.

How to configure it

The configuration has four parts: the synthetic test target, the blackbox probe, the probe store, and the Grafana dashboard.

Synthetic test target

A small service that increments a counter every 15 seconds. The service is a node_exporter-textfile collector, a shell script, or a tiny HTTP service.

#!/bin/bash
# /usr/local/bin/meta-test-emit.sh
# Run from cron or systemd timer every 15 seconds.
META_TEST_FILE=/var/lib/meta-test/meta_test_total.prom
mkdir -p "$(dirname "$META_TEST_FILE")"
NOW=$(date +%s)
echo "meta_test_total $NOW" > "$META_TEST_FILE"

The file is scraped by the production Prometheus using the node_exporter textfile collector, and federated to the meta. The meta should see meta_test_total incrementing every 15 seconds.

Blackbox probe

# /etc/meta-probe/blackbox.yml
modules:
  - name: prometheus_metrics_probe
    type: http_2xx
    http:
      method: GET
      url: https://prod-prom.internal:9090/metrics
      headers:
        Authorization: 'Basic meta-federation:REDACTED'
      fail_if_body_matches_regexp:
        - 'down'
      fail_if_header_not_matches:
        - header: content-type
          regexp: 'text/plain'
          allow_missing: false

The probe runs on a separate host. The probe store scrapes the blackbox exporter’s /metrics.

Probe store

# /etc/meta-probe/prometheus.yml
global:
  scrape_interval: 30s

scrape_configs:
  - job_name: blackbox
    static_configs:
      - targets: ['localhost:9115']

rule_files:
  - /etc/meta-probe/rules/probe.yml

alerting:
  alertmanagers: []  # No AM. The probe-alert path is email/SMS.

# /etc/meta-probe/rules/probe.yml
groups:
- name: probe
  rules:
  - alert: MetaEvidenceBroken
    expr: |
      probe_success{instance="prod-prom.internal:9090"} == 0
    for: 5m
    # No Alertmanager. The alert is delivered by the script
    # below, which reads the probe store's /api/v1/alerts.

#!/bin/bash
# /usr/local/bin/meta-evidence-pager.sh
# Run from cron every minute.
ALERTS=$(curl -s http://localhost:9090/api/v1/alerts \
  | jq -r '.data.alerts[] | select(.labels.alertname=="MetaEvidenceBroken") | .value')
if [ -n "$ALERTS" ]; then
  echo "Meta-evidence broken: production /metrics unreachable" \
    | mail -s "META-EVIDENCE BROKEN" oncall@example.com
fi

The script reads the probe store’s alerts and sends email via local MTA. The MTA is configured to relay via an external provider; the email delivery does not depend on Prometheus or the production stack.

Grafana dashboard

The dashboard has four row sections:

The top and middle rows read from the meta. The bottom row reads from the meta’s view of the synthetic test. The footer row reads from the probe store, not the meta. The footer is the only row that proves anything about the meta’s effectiveness.

How to validate it

Five checks confirm the meta-evidence is wired correctly.

# SEVERITY: READ-ONLY
# 1. Confirm the synthetic test target is emitting. Read the
#    textfile collector output.
cat /var/lib/meta-test/meta_test_total.prom

Expected output:

meta_test_total 1723651234

# SEVERITY: READ-ONLY
# 2. Confirm the meta sees the synthetic test counter
#    incrementing.
curl -s http://meta-prom:9090/api/v1/query \
  --data-urlencode \
  'query=delta(meta_test_total{prometheus_cluster="production"}[5m])' \
  | jq '.data.result[].value[1]'

A non-zero value confirms the federation path is working.

# SEVERITY: READ-ONLY
# 3. Confirm the blackbox probe is succeeding. Read the probe
#    store.
curl -s http://probe-store:9090/api/v1/query \
  --data-urlencode \
  'query=probe_success{instance="prod-prom.internal:9090"}' \
  | jq '.data.result[].value[1]'

A value of 1 confirms the probe can reach the production /metrics endpoint.

# SEVERITY: READ-ONLY
# 4. Confirm the Grafana dashboard has the four rows. The
#    footer row should be sourced from the probe store data
#    source, not the meta.
curl -s -u admin:$PASS http://grafana-meta:3000/api/dashboards/uid/meta-health \
  | jq '.dashboard.panels[] | select(.type=="row") | .title'

Expected output: four row titles matching the table above.

# SEVERITY: SERVICE-IMPACT (controlled test in maintenance window)
# 5. Run the controlled failure test. Stop the production
#    Prometheus for 5 minutes. Confirm three things:
#    a. The meta sees up == 0 on the production Prometheus.
#    b. The synthetic test counter stops incrementing (because
#       production is not scraping it any more).
#    c. The probe store's probe_success drops to 0.
#    d. The email alert arrives within 5 minutes.
systemctl stop prometheus
sleep 300
# Check each of a-d, then restart.
systemctl start prometheus

How it can fail

Six failure modes recur.

1. The synthetic test target lives on the production stack. Symptom: production is down, the synthetic test target is also down, the meta cannot tell whether the meta-evidence is broken or whether the production outage has cascaded. The target must be on a separate host.
2. The blackbox probe runs on the meta host. Symptom: meta is down, the probe is also down, the meta-evidence alert never fires. The probe must be on a separate host.
3. The probe store is the meta-Prometheus. Symptom: meta is down, the probe store is also down, the probe data is not stored. The probe store must be a separate Prometheus.
4. The Grafana dashboard reads only from the meta. Symptom: meta is down, dashboard renders empty, but the footer row that reads from the probe store is also empty because the dashboard data source was misconfigured to point at the meta. The footer row must read from a separate data source.
5. The meta-evidence alert routes through the meta Alertmanager. Symptom: meta is down, the alert is queued in the meta-AM, never delivered. The meta-evidence alert must route through a separate channel (email, SMS).
6. The dashboard has never been inspected by the on-call. Symptom: dashboard renders green for nine months because no one looks at it. The dashboard must be linked from the on-call runbook and inspected during incident drills.

How to troubleshoot it

When the meta-evidence dashboard shows an unexpected state, work from the probe outward.

1. Confirm the synthetic test target is running. systemctl status meta-test-emit.timer (or equivalent). If not running, restart and confirm the file is being written.
2. Confirm the production Prometheus is scraping the synthetic target. up{job="meta-test"} on the production Prometheus. If up == 0, the production Prometheus cannot reach the target; check the network.
3. Confirm the meta is federating the synthetic counter. Run delta(meta_test_total[5m]) on the meta. If the delta is zero, the federation selector is wrong.
4. Confirm the blackbox probe is reaching the production /metrics endpoint. probe_success on the probe store. If 0, the probe host cannot reach the production Prometheus.
5. Confirm the probe store is alerting. Inspect the probe store’s /api/v1/alerts endpoint. If the alert is missing, the rule is not firing.
6. Confirm the email is delivered. Check the local MTA logs. If the email is queued but not sent, the SMTP relay is the issue.
7. Form a hypothesis. The most common production failure is “synthetic test target on production stack” (row 1) or “blackbox probe on meta host” (row 2). Inspect the deployment topology.

Security implications

The probe store exposes the synthetic test counter and the blackbox probe metrics. These are not sensitive in themselves, but the probe store is on the management network and may be reachable from the production network for the blackbox probe. The probe store’s /metrics endpoint should be authenticated; the probe store’s alert delivery (email via local MTA) should not require credentials.

The email channel is a credential target. The MTA relay password is a secret; rotate it. The on-call email distribution list is itself an access-controlled resource; restrict who can add or remove members.

Performance implications

The synthetic test target is a shell script writing a small file. Negligible.

The blackbox probe is an HTTP request every 30 seconds. The production /metrics endpoint handles thousands of requests per second; the probe is a rounding error.

The probe store is a small Prometheus scraping one target every 30 seconds. Series count is in the low double digits. Disk usage is megabytes.

The email cron runs every minute. Mail delivery is a single SMTP transaction. Negligible.

The performance hazards are all about the operational cost of maintaining four components that must stay aligned. The cost is real but bounded.

Production guidance

• The synthetic test target is on a separate host from both the meta and the production stack. Three hosts in three failure domains.
• The blackbox probe is from a separate host. The probe host is not the meta host; not the production Prometheus host.
• The probe store is a separate Prometheus. Not a sidecar on the meta; not a sidecar on the production Prometheus.
• The meta-evidence alert routes through email or SMS, not PagerDuty. The channel is the only signal that works when the meta is down.
• The Grafana dashboard has a footer row that reads from the probe store. The footer is the only row that proves anything.
• Run the controlled failure test quarterly. The first test reveals a topology mistake the team did not know was there.
• Document the meta-evidence architecture in the on-call runbook. The on-call must know where to look when the meta is silent.

Verification

You should now be able to answer:

• What is meta-evidence, and why is it different from a meta-monitoring dashboard?
• What are the three components of meta-evidence, and why must each be on a separate host?
• Why must the meta-evidence alert route through email or SMS rather than PagerDuty through the meta Alertmanager?
• What is the controlled failure test for meta-evidence, and what does it prove?
• What is the first thing to check when the meta-evidence dashboard shows unexpected state?

Quiz