Federation for Meta

A team had two Prometheus servers scraping the same applications, each writing to its own TSDB. They called one “primary” and one “backup.” The primary was the one the dashboards read from. The backup was the one they hoped would still be running when the primary died. The two were independent. When the primary died, half the dashboards went grey, half the alerts stopped firing, and the “backup” was a separate TSDB with the same data the primary had been collecting up to ten minutes earlier. They called it “federation” because both servers “had the same data.” They had a parallel silo with a marketing name.

The right word for what they wanted is federation in the Prometheus sense: a label-matched pull of selected metrics from one Prometheus to another, over a dedicated /federate endpoint, on a schedule the receiving Prometheus controls. It is not a backup. It is not a parallel scrape. It is a stream of aggregates, scoped by label matchers, that the receiver stores as its own series. This lesson is about doing federation right for meta-monitoring.

What it is

Prometheus federation is a pull mechanism. A receiver Prometheus issues an HTTP GET against the source Prometheus’s /federate endpoint, passing label matchers as query parameters. The source Prometheus evaluates the matchers against its current series set and returns the matching series in the OpenMetrics text exposition format. The receiver parses the response, applies its own relabel rules, and stores the result as if it had scraped the series directly.

The mechanism is general — federation can be used for any Prometheus-to-Prometheus pull — but for meta-monitoring it has three specific properties that make it the right tool:

• The source does not need to know who is federating. Any Prometheus that has --web.enable-federation enabled exposes /federate. The source configuration is unchanged.
• The cardinality is bounded by the receiver’s matchers. A receiver that asks for match[]={job="prometheus"} receives only the Prometheus self-metrics. A receiver that asks for match[]={__name__=~"http_requests_total"} receives every http_requests_total series the source knows about — possibly millions. The receiver controls the scope.
• The data is aggregates, not full history. Each federation scrape returns the current value of each matching series. The receiver does not get the past; it gets the present. This is exactly what meta-monitoring wants: the current state of the production Prometheus, not a copy of its TSDB.

Why a sysadmin cares

The alternative to federation for meta-monitoring is remote write: the production Prometheus pushes its series to a long-term store, and the meta queries that store. Remote-write is the right tool for archival, long retention, and multi-tenant isolation. It is the wrong tool for meta-monitoring because the meta wants to control what it sees, not have the production stack decide. Federation lets the meta say “I want prometheus_tsdb_head_series aggregated by prometheus_cluster,” and the production Prometheus does the aggregation server-side.

Three operational reasons to use federation for the meta:

1. Cardinality control. The meta’s matchers are an explicit allow-list. A new application label in the production stack does not silently appear in the meta’s storage.
2. Server-side aggregation. The production Prometheus aggregates before sending. The meta receives one series per group, not millions of per-instance series.
3. No write coupling. Federation is a pull. The production Prometheus does not need to know the meta exists, does not queue writes for it, and does not block when the meta is down. The meta’s absence does not affect the production stack.

How it works

The flow is a single HTTP GET. The receiver builds the URL, the source evaluates it, the response is parsed as OpenMetrics text.

   Meta-Prometheus                        Production Prometheus
   ===============                        =====================

   GET /federate?match[]=...
        match[]={job="prometheus"}
        match[]={__name__="prometheus_tsdb_head_series"}
        match[]={__name__=~"prometheus_notifier_.*"}
   Authorization: Basic ...
   --->

                                          Evaluate match[] against
                                          current series set.
                                          Aggregate per group.
                                          Render as OpenMetrics text.

   <---
   200 OK
   # TYPE prometheus_tsdb_head_series gauge
   prometheus_tsdb_head_series{instance="prod-prom-1"} 1234567
   prometheus_tsdb_head_series{instance="prod-prom-2"} 1234890
   # TYPE prometheus_notifier_alerts_sent_total counter
   prometheus_notifier_alerts_sent_total{...} 42

   Parse response.
   Apply relabel_configs.
   Store as meta series with external_labels.

How to configure it

The configuration has two sides. The source Prometheus enables federation and (typically) adds recording rules for the metrics the meta will pull. The meta-Prometheus scrapes /federate with the right match[] parameters.

Source side (production Prometheus)

# /etc/prometheus/prometheus.yml (production)
global:
  scrape_interval: 15s
  evaluation_interval: 15s

# Enable /federate. By default it is disabled.
enable-federation: true

rule_files:
  - /etc/prometheus/rules/recording.yml
  - /etc/prometheus/rules/meta.yml

# No scrape config for the meta is needed. Federation is a pull;
# the meta initiates the connection.

The recording rules that produce meta-friendly aggregates:

# /etc/prometheus/rules/meta.yml
groups:
- name: meta.recording
  interval: 30s
  rules:
  - record: job:prometheus_http_requests:rate5m
    expr: |
      sum by (job, code) (
        rate(prometheus_http_requests_total[5m])
      )
  - record: job:prometheus_tsdb_head_series:max
    expr: |
      max by (job, instance) (
        prometheus_tsdb_head_series
      )
  - record: job:prometheus_notifier_alerts:rate5m
    expr: |
      sum by (job, integration) (
        rate(prometheus_notifier_alerts_sent_total[5m])
      )

These three recording rules cover the three signal classes the meta cares about: scrape activity (request rate and HTTP codes), storage state (head-block series count), and alert dispatch (alert-send rate by integration).

Receiver side (meta-Prometheus)

# /etc/meta-prometheus/prometheus.yml
scrape_configs:
  - job_name: prod-prom-federate
    scheme: https
    metrics_path: /federate
    params:
      match[]:
        # Self-metrics of the production Prometheus itself.
        - '{job="prometheus"}'
        # Recording-rule outputs for the meta to consume.
        - '{__name__="job:prometheus_http_requests:rate5m"}'
        - '{__name__="job:prometheus_tsdb_head_series:max"}'
        - '{__name__="job:prometheus_notifier_alerts:rate5m"}'
    basic_auth:
      username: meta-federation
      password_file: /etc/meta-prometheus/secrets/federation.pass
    static_configs:
      - targets: ['prod-prom-1.internal:9090']
        labels:
          prometheus_cluster: production

Reading the config:

• metrics_path: /federate — the meta is hitting the federation endpoint, not /metrics. /metrics would return every series; /federate evaluates the matchers.
• params.match[] is a list. Each entry is one selector. The meta receives the union. Keep the list short — every entry is a query against the production Prometheus on every scrape.
• The first selector is {job="prometheus"}, which returns all series with job="prometheus". The production Prometheus labels its own self-metrics with job="prometheus", so this is a clean way to grab the lot.
• The recording rule outputs use the __name__ matcher. They pre-aggregate server-side, so the meta receives one series per label group, not one per scrape target.
• basic_auth with a password_file, not a literal. The credential is for the production Prometheus --web.config htpasswd entry, scoped to read-only on /federate.

How to validate it

Five checks confirm the federation pull is working as designed.

# SEVERITY: READ-ONLY
# 1. Confirm /federate is enabled on the source.
curl -s https://prod-prom:9090/federate?match[]=%7Bjob%3D%22prometheus%22%7D \
  -u meta-federation:$PASS \
  | head -20

Expected output:

# TYPE prometheus_build_info gauge
prometheus_build_info{branch="HEAD",goarch="amd64",...} 1
# TYPE prometheus_config_last_reload_success_timestamp_seconds gauge
prometheus_config_last_reload_success_timestamp_seconds 1723651234.567
# TYPE prometheus_http_requests_total counter
prometheus_http_requests_total{code="200",handler="/metrics"} 12345
...

# SEVERITY: READ-ONLY
# 2. Confirm /federate is NOT exposed without authentication.
curl -s -o /dev/null -w '%{http_code}' \
  https://prod-prom:9090/federate?match[]=%7Bjob%3D%22prometheus%22%7D

Expected output: 401. If the response is 200, the production Prometheus is serving /federate to anyone who can reach the port. Fix this before continuing.

# SEVERITY: READ-ONLY
# 3. Confirm the meta is receiving the recording-rule output.
curl -s http://meta-prom:9090/api/v1/query \
  --data-urlencode \
  'query=job_prometheus_http_requests_rate5m{prometheus_cluster="production"}' \
  | jq '.data.result[].metric'

The result should show one series per code label group (e.g., code="200", code="500"). If the result shows series per scrape target, the recording rule on the source is wrong.

# SEVERITY: READ-ONLY
# 4. Confirm the meta scrape is reaching the source.
curl -s http://meta-prom:9090/api/v1/query \
  --data-urlencode \
  'query=up{job="prod-prom-federate",cluster="meta"}' \
  | jq '.data.result[] | .value[1]'

A result of 1 means the meta reached /federate, got a 200, and is actively scraping. A result of 0 means the scrape is failing; inspect the meta logs for the exact error.

# SEVERITY: READ-ONLY
# 5. Confirm the meta is NOT receiving application series.
curl -s http://meta-prom:9090/api/v1/query \
  --data-urlencode \
  'query=count(up{cluster="meta",job=~"app|service|checkout|payments"})' \
  | jq '.data.result'

An empty result is correct. If non-empty, the federation selectors have leaked into application label space and the meta is becoming a parallel silo.

How it can fail

Five failure modes recur across production federation setups.

1. /federate exposed without auth. Symptom: a curl from any host on the management network returns 200 with the full series set. Anyone who can reach the port can pull every Prometheus self-metric, including alertmanager integration URLs that may contain tokens. Fix: basic_auth and an authenticating reverse proxy.
2. Selectors too broad. Symptom: the meta’s series count grows to match the production Prometheus. The meta becomes the parallel silo the discipline exists to prevent. Fix: reduce match[] to the self-metrics and recording-rule outputs the meta actually needs.
3. Selectors too narrow. Symptom: the meta dashboards show empty panels because the selectors did not match anything. The __name__ matcher is case-sensitive; a typo means zero matches. Fix: validate selectors by running them against the production Prometheus query API first.
4. No aggregation on the source. Symptom: the meta’s storage grows linearly with the production Prometheus’s label cardinality. A new high-cardinality label in production silently explodes the meta’s TSDB. Fix: pre-aggregate with recording rules on the source.
5. Federation used for everything. Symptom: the team treats federation as a backup mechanism and expects the meta to contain a copy of the production TSDB. The meta’s storage grows past its provisioned size and Prometheus starts dropping samples. Fix: use federation for selected aggregates; use remote-write for full archival if that is what the team needs.

How to troubleshoot it

Work the request from the meta host outward.

1. Confirm the meta is configured to scrape /federate. grep metrics_path /etc/meta-prometheus/prometheus.yml. The path is /federate, not /metrics.
2. Confirm the meta is reaching the source. up{job="..."} on the meta. If up == 0, the HTTP request failed; check the meta logs for the status code.
3. Confirm the source is responding. From the meta host, curl -u meta-federation:$PASS https://prod-prom:9090/ federate?match[]=.... The response should be a 200 with OpenMetrics text.
4. Confirm the selectors match something. Run the same match[] against the production Prometheus /api/v1/query endpoint. If the query API returns zero series, the federation call will too.
5. Confirm the recording rules exist on the source. If the meta is federating __name__="job:..." and getting empty results, the recording rule that produces that name does not exist on the source.
6. Confirm the meta’s relabel rules. A metric_relabel_configs block that drops the wrong metric name will silently disappear from the meta. Comment out the relabel block and re-scrape; if the series appear, the relabel is the bug.

Security implications

The /federate endpoint is a read-everything interface. With no match[] filter, it returns every series the source knows. With a broad match[], it returns a substantial subset. The endpoint also exposes label values that name hosts, services and tenants — the same label values the source’s own dashboards show.

Production mitigations, in order of impact:

• Authenticate. basic_auth with a read-only service account, or mTLS. Production deployments should never serve /federate without auth.
• Restrict the network. The /federate endpoint should be reachable only from the meta-Prometheus host or the management network segment. A network ACL or firewall rule on the production Prometheus host.
• Restrict the selectors. The meta’s match[] is the implicit allow-list. Anything outside it cannot be read. Keep the list short and explicit.
• Audit the meta’s storage. The meta is a smaller copy of the production data. Treat the meta’s TSDB with the same sensitivity as the production TSDB. Restrict who can query the meta.
• Rotate the federation credential. The same rotation cadence as any other service account. Federation credentials have a way of being pasted into wiki pages.

Performance implications

Federation is cheap on the receiver side and bounded on the source side. The receiver issues one HTTP GET per scrape, the source evaluates the selectors and returns the result. The network cost is the size of the response, which is bounded by the selectors.

The cost is the storage on the receiver. A federation selector that returns a million series means the meta-Prometheus stores a million series. With a thirty-day retention, that is significant disk. The fix is server-side aggregation on the source: pre-aggregate with recording rules, then federate the rule outputs. The meta stores one series per label group, not one series per scrape target.

The scrape interval on the meta is the latency floor. A fifteen-second scrape interval means the meta is fifteen seconds behind the source. For meta-monitoring, this is acceptable. For alerting on a per-target basis, it is not — the meta is not the right place for that, and the team should be using the production Prometheus directly.

Production guidance

• Enable /federate on the source with --web.enable-federation or enable-federation: true in the YAML.
• Authenticate the endpoint. Production /federate is not public.
• Pre-aggregate on the source. Recording rules that produce job:metric:aggregation outputs are the right shape for the meta to consume.
• Keep the meta’s match[] short and explicit. Every entry is a query against the source on every scrape.
• Validate the selectors against the source’s /api/v1/query endpoint before deploying.
• Treat the meta’s TSDB as sensitive data. The same labels that appear in the production dashboards appear in the meta.

Verification

You should now be able to answer:

• What is Prometheus federation, and how does it differ from remote-write?
• Which three properties of federation make it the right tool for meta-monitoring?
• Why must /federate be authenticated in production?
• What is the right shape for the meta’s match[] selectors?
• What is the first thing to check when the meta dashboards show empty panels after a federation configuration change?

Quiz