Avoiding Circular Assumptions

A page should fire at 02:14. The on-call engineer should receive it at 02:15. The mitigation should start at 02:16. None of that happened, because the meta-Prometheus was hosted on the same Kubernetes cluster as the production stack it was monitoring, and the cluster’s API server was the broken thing.

The dependency chain:

   Production stack         Cluster API       Meta stack
   ===============         ===========       ==========

   Prometheus
   Alertmanager
   Loki ingesters
   Tempo distributors
        |
        |  (deployments, ConfigMaps,
        |   Services, Ingress)
        v
                          K8s API server
                                |
                                | (meta-pod runs here,
                                |  meta-ConfigMap lives here)
                                v
                          Meta-Prometheus
                          Meta-Alertmanager

When the API server went down at 02:14, the meta-Prometheus pod could not refresh its service account token. The pod stayed running but its federated scrape failed. The meta could not evaluate its rules. The meta-Alertmanager, also in the same cluster, also could not authenticate against its own configuration. The notification destination never received the MetaPrometheusDown page. The on-call engineer learned about the outage from a customer at 02:31.

The root cause was not the API server. The root cause was the decision to put the meta inside the cluster it was monitoring. This lesson is about the discipline of seeing those decisions before they happen.

What it is

A circular assumption in meta-monitoring is a deployment where the meta’s ability to alert on a production failure depends on the production stack itself being healthy. The meta is supposed to be the last line of defence; when the meta shares state with the production stack, the last line of defence has the same blast radius as the thing it is defending.

The dependency can be direct (the meta uses a service the production stack provides) or indirect (the meta and the production stack share an upstream dependency). Both are circular; both defeat the meta.

The discipline is to enumerate every shared dependency and eliminate it before declaring the meta complete. The enumeration is a static exercise on a whiteboard; the elimination is operational work.

Why a sysadmin cares

The cost of a circular dependency is the cost of the failure it masks. A meta that cannot detect a production outage is not a meta; it is decoration. The team pays the cost of running it, gets no operational signal from it, and discovers the gap during the next production incident. The discovery is embarrassing and the post-mortem is unsparing.

The shared dependencies recur across every meta-monitoring deployment. The eight most common:

The first six are easy to miss because each one looks infrastructure-y rather than production-y. The team did not think of the LDAP outage as a meta-monitoring failure; they thought of it as an authentication failure. The post-mortem forced the rethink.

How it works

The mental model is a dependency graph. The meta is the subject; the production stack is the object being monitored. Every edge in the graph that connects the meta to a service that is also depended on by the production stack is a candidate circular dependency.

                            +-- Production Prometheus
                            |
   Time source (NTP) -------+-- Meta-Prometheus
                            |
                            +-- Production Alertmanager
                            +-- Meta Alertmanager

                            +-- LDAP / OIDC
                            |
   Auth backend -------------+-- Production Prometheus
                            |
                            +-- Meta-Prometheus
                            +-- (any service that
                                authenticates users)

                            +-- PagerDuty / Slack
                            |
   Notification -------------+-- Production Alertmanager
   destination               |
                            +-- Meta Alertmanager

The graph makes the circularity obvious when drawn. The operational discipline is to draw it before deploying the meta, not after the post-mortem.

How to configure it

The configuration that prevents circular dependencies is mostly a deployment-shape decision, not a YAML change. The YAML changes that matter are the credential sources, the network ranges, and the notification webhook endpoints.

# /etc/meta-prometheus/prometheus.yml
global:
  # Hard-coded IP, not DNS. If the production DNS is down, the
  # meta still reaches the production Prometheus. Hard-coding
  # the meta's targets is a deliberate separation discipline.
  external_labels:
    cluster: meta

scrape_configs:
  - job_name: prometheus
    scheme: https
    # Credentials read from a local file on the meta host. The
    # meta does not call out to Vault to retrieve them. A Vault
    # outage cannot break the meta scrape.
    basic_auth:
      username: meta-scraper
      password_file: /etc/meta-prometheus/secrets/prod-prom.pass
    static_configs:
      - targets: ['10.40.1.10:9090']
        labels: { prometheus_cluster: production }

alerting:
  alertmanagers:
    # Static IPs again. The meta-Alertmanager is on a known
    # IP range that the meta can reach even if production DNS
    # is broken.
    - static_configs:
        - targets: ['10.50.1.10:9093', '10.50.1.11:9093']

# /etc/meta-alertmanager/alertmanager.yml
route:
  receiver: meta-platform-pager

receivers:
  - name: meta-platform-pager
    pagerduty_configs:
      # Routing key from a local file. The meta does not
      # retrieve the routing key from Vault at runtime.
      - routing_key:
          from_file: /etc/meta-alertmanager/secrets/pager.key
        severity: 'critical'

Reading the configs:

• Hard-coded IPs for both the scrape targets and the Alertmanager targets. The meta does not depend on DNS to reach the things it must reach.
• Credentials read from local files, not retrieved from a secrets manager at scrape time. The meta can run without Vault.
• The notification routing key is also from a local file. A Vault outage cannot stop the meta from paging.

How to validate it

Five checks confirm the absence of circular dependencies.

# SEVERITY: READ-ONLY
# 1. Confirm the meta-Prometheus host does not depend on
#    production DNS. From the meta host, resolve a known
#    production hostname.
dig +short prod-prometheus.internal

A non-empty answer is fine; the test is whether the meta host requires the answer. Check the meta’s scrape config for hard-coded IPs rather than hostnames.

# SEVERITY: READ-ONLY
# 2. Confirm the meta's credentials are local, not from a
#    vault. Inspect the meta's startup configuration.
grep -E 'password_file|api_key_file|routing_key' \
  /etc/meta-prometheus/prometheus.yml \
  /etc/meta-alertmanager/alertmanager.yml

Every credential should be a *_file reference, not an inline value or a remote-fetch URL.

# SEVERITY: READ-ONLY
# 3. Confirm the meta Alertmanager is on a separate cluster.
#    The peer list should not contain any production
#    Alertmanager peers.
curl -s http://meta-alertmanager:9093/api/v1/status \
  | jq '.data.cluster.peers[]'

The list should contain only meta-Alertmanager instances.

# SEVERITY: READ-ONLY
# 4. Confirm the meta is not on the same cluster as the
#    production stack. If you run Kubernetes, the meta pods
#    should be in a separate cluster, or at minimum a separate
#    node pool with no production workloads.
kubectl --context=meta get pods -A -o wide \
  | grep -v kube-system
kubectl --context=prod get pods -A -o wide \
  | grep -v kube-system

The two outputs should have no overlapping nodes.

# SEVERITY: SERVICE-IMPACT (controlled test in maintenance window)
# 5. Run the controlled failure test. Stop the production
#    Prometheus for 5 minutes. Confirm the meta still pages.
systemctl stop prometheus    # on production host
sleep 300
# After 5 minutes, the meta should have fired
# MetaPrometheusDown and the page should be at the on-call.
# Restart the production Prometheus.
systemctl start prometheus

A page that arrives during the 5-minute window confirms the end-to-end path is free of circular dependencies. An absent page means one of the eight shared dependencies is still in place.

How it can fail

Six circular-dependency failure modes recur.

1. Meta on the same K8s cluster as production. Symptom: the cluster API server is down, both the production stack and the meta are degraded, and the meta cannot alert about either. The post-mortem names “shared cluster” as the root cause.
2. Meta uses production LDAP for basic_auth credentials. Symptom: LDAP is down, the meta cannot authenticate against the production Prometheus /metrics, and the meta cannot tell that the production Prometheus is up because the scrape fails with 401.
3. Meta fetches credentials from Vault at scrape time. Symptom: Vault is down, the meta Prometheus cannot start because it cannot read its password_file, and the meta process exits. The meta is silent during the Vault outage.
4. Meta-Alertmanager pulls routing keys from Vault. Symptom: Vault is down, the meta-Alertmanager cannot push to PagerDuty, and the meta alerts about the production outage never reach the on-call.
5. Meta uses DNS to resolve production hostnames. Symptom: production DNS is down, the meta cannot resolve the production Prometheus hostname, and the meta cannot scrape the production stack. Hard-coded IPs would have prevented this.
6. Meta shares NTP with production. Symptom: NTP source is unreachable, the meta and production clocks drift apart, and the meta dashboards show the production data as “5 minutes ago” when it is actually current. The investigation chases a non-existent delay.

How to troubleshoot it

When the meta is silent during a production outage, the question is which shared dependency is the cause. Work the eight dependencies in order of failure-mode severity.

1. Confirm the meta process is up. If down, the host OS or cluster is the shared dependency. Inspect the meta host’s kernel logs and the cluster’s control-plane status.
2. Confirm the meta host can reach the meta Alertmanager. If not, the network is shared. Inspect routing tables and switch state.
3. Confirm the meta can authenticate to its scrape targets. If 401 errors appear, the auth backend is shared. Check the meta’s credentials and the auth backend’s status.
4. Confirm the meta can resolve its scrape targets. If DNS errors appear, the DNS is shared. Replace hostnames with IPs in the meta’s scrape config.
5. Confirm the meta-Alertmanager can reach the notification destination. If webhook errors appear, the secrets manager is shared. Move the routing key to a local file.
6. Confirm the meta and production clocks are within one second. If not, NTP is shared. Configure the meta host with a different NTP source.
7. Confirm the meta Grafana can render the production data. If not, the Grafana data source is shared. Point the meta Grafana at the meta Prometheus.
8. Form a hypothesis. The most common production circular dependency is the cluster (row 1) or the auth backend (row 2). Inspect the meta logs for the exact failure.

Security implications

The circular-dependency discipline interacts with the security discipline in two places.

• Credential storage. Local credential files on the meta host are simpler than Vault retrieval but require filesystem permissions. The meta host’s /etc/meta-prometheus/secrets/ directory must be 0700, owned by the meta-prometheus user. A compromise of the meta host is a compromise of the meta’s credentials.
• Network exposure. Hard-coded IPs across network ranges means the meta’s targets are static. A firewall rule on the meta host that restricts outbound traffic to the production IP ranges only is the right shape. The meta should not have a route to the public internet except for the notification destination.

Performance implications

Hard-coded IPs and local credentials are not performance hazards. They remove a network round-trip and a vault round-trip respectively, both of which are wins.

The performance hazard is the meta’s own clock skew. If the meta host’s clock drifts, the meta’s timestamps on alerts and metrics are wrong. The on-call sees “the alert fired at 02:14” when the meta actually evaluated the rule at 03:14. The investigation follows the wrong timeline.

Production guidance

• Draw the dependency graph before deploying the meta. Whiteboard exercise; ten minutes; saves months of post-mortems.
• Hard-code IPs in the meta’s scrape and alerting configs. DNS is a shared dependency.
• Local credential files, not Vault retrieval. Vault retrieval is a shared dependency.
• Separate NTP source for the meta host, or a separate NTP path that does not depend on the production network.
• Separate Kubernetes cluster for the meta, or at minimum a separate node pool with no production workloads.
• Run the controlled failure test quarterly. The first test almost always reveals a circular dependency the team did not know was there.

Verification

You should now be able to answer:

• What is a circular dependency in meta-monitoring, and how is it different from a shared infrastructure dependency?
• Which eight shared dependencies recur across meta-monitoring deployments?
• Why must the meta use hard-coded IPs rather than DNS for its scrape and alerting targets?
• What is the controlled failure test, and what does it prove?
• What is the first thing to check when the meta is silent during a real production outage?

Quiz