Separate Monitoring Platform

A post-mortem from a regional bank, three years ago: “the Alertmanager cluster lost quorum at 02:11 due to a network partition affecting the monitoring VLAN. Production alerts from the same cluster were already in flight. The meta-Prometheus was running on a worker node in the same Kubernetes cluster. The meta was silent until 03:47 when an engineer manually re-routed DNS. Root cause: shared failure domain.”

The lesson is that the meta-Prometheus has to be separate, not just additional. “A second Prometheus on the same cluster” is not a meta-monitoring platform. It is a parallel silo that inherits every shared-fate failure mode of the production stack. This lesson is about the discipline of separation: what to split, what to keep distinct, and how to prove the separation holds.

What it is

A separate monitoring platform is a Prometheus deployment that shares no infrastructure with the production stack it watches. Different hosts, ideally different network segments, distinct storage, distinct credentials, distinct alerting path. The production stack scrapes applications; the meta stack scrapes the production stack. The two stacks meet only at the network boundary, over an authenticated channel.

The alternative — a second Prometheus on the same hosts, the same network, the same database — is a parallel silo. It has the same blast radius as the production stack, the same failure modes, the same shared dependencies. When the production stack is down, the silo is also down. The silo cannot warn you about the outage because it is the outage.

Why a sysadmin cares

The shared-fate failure modes that the separate-platform design removes are not theoretical. They are the failure modes that have taken down production observability stacks in the wild.

   Failure mode                       Co-located meta  Separate meta
   ---------------------------------  ---------------- ----------------
   Production host OS crashes         Silent            Still scraping
   Production K8s node cordoned       Silent            Still scraping
   Production network VLAN down       Silent            Still scraping
   Production auth backend (LDAP)     Cannot auth       Local accounts
   down                                on /metrics
   Production Alertmanager cluster    Alerts unsent     Meta has own AM
   loses quorum
   Production DNS resolution broken   Cannot resolve    Hard-coded IPs
                                      production hosts   or own DNS
   Production storage backend down    Both TSDBs dead   Meta has own
   (shared object store)
   Production certificate expired     Both TLS-broken   Meta has own cert

Every row in the right column is a real incident class. The separate platform does not have to prevent every one of them; it has to prevent the class where the meta is the only thing that could have detected the production failure and is also the thing that fails.

How it works

The separation runs along five axes. Each axis is a place where the meta and the production stack could share state, and a place where that sharing has to be removed.

   Axis         Production stack         Meta stack
   -----------   ----------------------   ----------------------
   Hosts        prod-prom hosts          meta-prom hosts
   Network      monitoring VLAN          separate management VLAN
   Storage      prod TSDB (object store) meta TSDB (different bucket
                                          or local SSD)
   Credentials  prod service accounts    meta service accounts
   Alerts       prod Alertmanager        meta Alertmanager

Each axis can be relaxed slightly, but each relaxation is a trade you make deliberately. The default for production is maximum separation on every axis.

How to configure it

The configuration that enforces separation is the production scrape config from lesson 01, with two additions: distinct external_labels and a local-disk storage path the production stack does not touch. The path is a startup flag, not a prometheus.yml key:

# /etc/default/meta-prometheus
# Local SSD. The meta-Prometheus does not write to the
# production object store; that would re-introduce the
# storage axis as a shared dependency.
ARGS="--storage.tsdb.path=/var/lib/meta-prometheus"

# /etc/meta-prometheus/prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    # Distinguishes meta series from production series in any
    # downstream system that sees both (Grafana, Alertmanager,
    # long-term storage).
    cluster: meta
    environment: management
    replica: meta-prom-1

scrape_configs:
  - job_name: prometheus
    scheme: https
    basic_auth:
      username: meta-scraper
      password_file: /etc/meta-prometheus/secrets/prod-prom.pass
    static_configs:
      - targets:
          - 10.40.1.10:9090
          - 10.40.1.11:9090
        labels:
          prometheus_cluster: production

  - job_name: alertmanager
    static_configs:
      - targets: ['10.40.2.10:9093', '10.40.2.11:9093']
        labels: { component: alertmanager }

  - job_name: grafana
    static_configs:
      - targets: ['10.40.3.10:3000']
        labels: { component: grafana }

rule_files:
  - /etc/meta-prometheus/rules/*.yml

alerting:
  alertmanagers:
    - static_configs:
        # The meta Alertmanager is on the management network,
        # not the monitoring VLAN. Even if the monitoring VLAN
        # is partitioned, meta alerts can still reach it.
        - targets: ['10.50.1.10:9093', '10.50.1.11:9093']

Reading the config:

• external_labels.cluster: meta — every series the meta-Prometheus produces carries this label. A query that joins cluster="meta" and cluster="production" in the same panel is a misconfiguration; the meta is not supposed to produce production series.
• storage.path is local SSD, not the production object store. The meta’s TSDB lives on the meta host’s disk. Backups go to a separate backup bucket.
• basic_auth is for the production Prometheus self-metrics endpoint. The meta uses a distinct service account, with distinct credentials, in the production Prometheus’s --web.config file.
• The meta Alertmanager targets are on a different network range. The DNS name meta-alertmanager does not resolve from the monitoring VLAN; the monitoring VLAN’s DNS does not resolve from the management VLAN. The split is at the network layer.

How to validate it

Four checks confirm the separation holds.

# SEVERITY: READ-ONLY
# 1. Confirm the meta-Prometheus is on a different host. The
#    production Prometheus's node_exporter exposes the hostname
#    in node_uname_info; the meta should not appear there.
curl -s http://prod-prom:9090/api/v1/query \
  --data-urlencode \
  'query=count(node_uname_info{cluster="production"}) by (nodename)' \
  | jq '.data.result[].metric.nodename'

The list should contain the production Prometheus host names, not the meta-Prometheus host names.

# SEVERITY: READ-ONLY
# 2. Confirm the meta-Prometheus has its own external_labels.
#    Every meta series should carry cluster="meta".
curl -s http://meta-prom:9090/api/v1/query \
  --data-urlencode 'query=count(up) by (cluster)' \
  | jq '.data.result[] | {cluster: .metric.cluster, count: .value[1]}'

Expected output:

{ "cluster": "meta", "count": "5" }

If cluster="production" appears in the meta’s own query result, the meta is scraping applications it should not be.

# SEVERITY: READ-ONLY
# 3. Confirm the meta Alertmanager is on a different network.
#    From the meta host, the meta Alertmanager is reachable
#    and the production Alertmanager is not (or is, but only
#    over a path the production stack does not use).
ss -tn dst ':9093' | sort -u

The list should show the meta Alertmanager targets in the 10.50.1.0/24 range and not the production Alertmanager targets in the 10.40.2.0/24 range. If both ranges appear, the meta host has network visibility into both networks and the separation is at the process layer only.

# SEVERITY: READ-ONLY
# 4. Confirm the meta has its own storage path.
ls -ld /var/lib/meta-prometheus
stat -c '%U %a' /var/lib/meta-prometheus

The directory should be owned by the meta-prometheus user (or its systemd DynamicUser equivalent), with mode 0700 or 0750.

How it can fail

Six failure shapes recur in production. Each one is a place where the meta inherits a shared-fate dependency from the production stack.

1. Same hosts, different processes. The meta-Prometheus runs in a sidecar on the production Prometheus host. Symptom: host crash takes both layers down; no separation in the host axis. Often deployed “because we have spare CPU on the monitoring host.” Wrong trade.
2. Same K8s cluster, different namespace. The meta runs in namespace: meta, the production stack runs in namespace: monitoring. Symptom: the cluster’s API server is down, so both stacks cannot scrape or alert; node cordon on the worker pool takes both down.
3. Same object store. The meta writes its TSDB to the same S3 bucket the production Prometheus remote-writes to. Symptom: bucket policy change, expired credential, or region outage takes both layers down. The meta’s storage axis is silently shared.
4. Same alerting path. The meta alerts through the production Alertmanager cluster. Symptom: production Alertmanager loses quorum, meta alerts about the quorum loss are also unsent. The meta’s alerting axis is shared.
5. Same auth backend. The meta uses the production LDAP for its own user accounts and for the basic_auth it uses to scrape the production stack. Symptom: LDAP outage prevents the meta from authenticating; meta alerts cannot even be read.
6. Same network. The meta and the production stack are on the same VLAN. Symptom: a switch reboot during a firmware upgrade takes both layers down. The meta’s network axis is shared.

How to troubleshoot it

When the meta is silent and the production stack is also silent, work the layers in the order they share state.

1. Confirm the meta process is up. systemctl status meta-prometheus. If it is up, the meta is running but failing somewhere downstream.
2. Confirm the meta host is reachable. If the meta host is also down, the shared-fate failure is at the host axis.
3. Confirm the meta host’s network reachability. From the meta host, can you reach the production Alertmanager? If yes, the network axis is shared. The meta should only reach the meta Alertmanager.
4. Confirm the meta’s own storage. df -h /var/lib/ meta-prometheus. If the disk is full or the mount is gone, the meta will not start its WAL replay.
5. Confirm the meta’s auth path. If the meta uses LDAP and LDAP is down, the meta cannot scrape its basic_auth-protected targets. Check getent passwd meta-scraper or equivalent.
6. Form a hypothesis. The most common production failure is “shared network” or “shared auth backend.” Inspect the meta logs for the exact scrape error; the meta is the source of truth for the meta’s own state.

Security implications

The separate platform has a smaller blast radius than the production stack but a higher-value blast radius. A compromise of the meta tells the attacker the state of the entire production stack. The meta should be hardened as if it were production:

• Restrict the meta query API to the management network.
• Authenticate all scrapes; the meta is what is scraped, so restrict who can scrape the meta.
• Audit meta alert routes. The meta pages the on-call rotation; the route table is the table an attacker would want to subvert.
• Do not co-locate meta credentials with production credentials in the same secret manager or vault path. A blast radius that hits the secret manager should not hit both layers.

Performance implications

The separate platform adds cost. The meta is roughly one VM (2 vCPU, 4 GB RAM, 50 GB SSD), one Alertmanager cluster (two small VMs), and the operational cost of keeping the meta’s configuration, rules, and retention current.

The cost is fixed and small. The cost of not having it is whatever the next production observability outage costs. The cost-benefit calculation favours the meta in any environment that pages on production incidents.

Production guidance

• Order the separation axes by blast radius. Hosts first, then network, then storage, then credentials, then alerting. Spend the budget on the first two before the others.
• A meta-Prometheus on the same hosts as production is not meta-monitoring. It is decoration.
• A meta-Prometheus on different hosts but the same K8s cluster is also decoration. The cluster is the host.
• Use distinct external_labels on the meta. The cluster="meta" label is what makes the meta’s data recognisable downstream.
• Do not co-locate meta credentials with production credentials. Different secret manager paths, different service accounts, different rotation schedules.
• Back up the meta’s TSDB. A meta you cannot restore from backup is a meta you cannot trust after a disk failure.

Verification

You should now be able to answer:

• What are the five axes along which the meta and production stacks must be separate?
• Which of those axes are the most important to separate first?
• What is a parallel silo, and how is it different from a meta-monitoring platform?
• What is the first thing to check when the meta is silent and the production stack is also silent?
• Which shared-fate failure mode does a “same K8s cluster, different namespace” meta inherit?

Quiz