Federation

Six Kubernetes clusters, six Prometheus servers, and a request from management for “one dashboard for the whole platform”. The classic Prometheus answer is hierarchical federation: a global server that scrapes a small, curated subset of series from each leaf. It is a real, working design — and it has hard limits that the ecosystem eventually built Thanos, Mimir and remote write to route around. Knowing where the limits are is most of this lesson.

What it is

Every Prometheus server exposes a /federate HTTP endpoint. When something scrapes it, the server evaluates the match[] URL parameters as instant-vector selectors against its own TSDB and returns the current values in exposition format. A higher-level Prometheus scrapes /federate exactly like any other target, and the selected series are copied into its own TSDB.

The mental model: federation is just a scrape where the target’s /metrics output happens to be computed by a query. Everything from the scrape lifecycle lesson applies — intervals, timeouts, relabeling, and up{job="federate"} as the liveness signal of the chain itself.

 leaf Prometheus (dc1)            leaf Prometheus (dc2)
  scrapes its nodes                scrapes its nodes
        |                                |
        +----  /federate?match[]=...  ---+
                     \              /
                      v            v
                    global Prometheus
                 (job: federate, honor_labels: true)
                            |
                         Grafana

How it is used well

The pattern that survives contact with production:

1. Leaf servers run recording rules that pre-aggregate to the level the global view needs — per-job or per-cluster ratios and rates, not per-container raw series.
2. The global server federates only those aggregates plus a handful of small series such as up.
3. Leaves carry external_labels (for example dc: eu-west-1) so the same aggregate from two datacentres stays distinguishable.

That is a few hundred to a few thousand series on the global server. Not the firehose.

honor_labels: true on the federation job is what makes this work. Without it, the global server rewrites job and instance to the federation job’s own values (the originals survive as exported_job / exported_instance), and series from two leaves collide into identical label sets.

On match[]: each parameter is one selector, and each selector costs the leaf one instant query per scrape. Select several metrics at once with a regex on __name__, and keep the number of selectors small.

The honest limits

• Copies, not views. Every federated series is stored again in the global TSDB, at the federation scrape interval’s resolution. Federation doubles storage for whatever it carries.
• No backfill. If the global server is down for an hour, that hour is a permanent gap. There is no replay mechanism.
• Staleness passes through bluntly. When a series disappears on a leaf, it disappears from /federate, and the global server marks it stale at its own pace. Native histograms and exemplars do not cross federation in 2.55.
• No global query over raw data. Federating raw high-cardinality metrics is the classic way to melt a Prometheus server. If the requirement is “query any container metric across all clusters”, federation is the wrong tool.
• No multi-tenancy, no access-control story.

When those limits bite, the alternatives are lesson 06 territory: remote write into Thanos Receive, Mimir, Cortex or VictoriaMetrics gives you global queries over raw series, HA-pair deduplication, object-storage retention and multi-tenancy. Federation remains legitimate for curated aggregates, low-churn estates, and as a pragmatic bridge while the real platform is built.

How to configure it

On the global server:

scrape_configs:
  - job_name: federate-dc1
    honor_labels: true            # keep the leaf's job/instance labels
    metrics_path: /federate
    params:
      match[]:
        # pre-aggregated recording rules plus up; nothing raw
        - '{__name__=~"job:.*:ratio|job:.*:rate5m"}'
        - 'up'
    static_configs:
      - targets: ['prometheus.dc1.internal:9090']
    scrape_interval: 60s
    scrape_timeout: 30s           # /federate does real query work; give it room

The leaf needs no federation-specific configuration at all — only the recording rules that produce the series being selected, and external_labels distinguishing it from its siblings.

How to validate it

# See exactly what the global server will ingest (note curl -g for the brackets)
curl -g -s 'http://prometheus.dc1.internal:9090/federate?match[]=up' | head
# # TYPE up untyped
# up{instance="node-11.dc1:9100",job="node",dc="eu-west-1"} 1
# ...

# Is the chain alive, and how expensive is it?
curl -s 'localhost:9090/api/v1/query?query=up%7Bjob%3D%22federate-dc1%22%7D'
curl -s 'localhost:9090/api/v1/query?query=scrape_duration_seconds%7Bjob%3D%22federate-dc1%22%7D'

Then sanity-check the global TSDB: do the federated series carry the leaf’s original job/instance plus the expected dc label, and does the series count match what the selector returns on the leaf? If the global count is far higher than the leaf’s /federate output, something upstream is double-scraping.

How it can fail

1. A match[] too broad. Someone federates a selector matching half the leaf’s TSDB; the instant query outruns scrape_timeout, up{job="federate-…"} flaps, and the leaf’s CPU spikes every interval. The most common federation incident.
2. honor_labels left false. Series from two leaves collapse into identical label sets; symptoms are “duplicate sample for timestamp” errors in the global server’s logs, or series that oscillate between two datacentres’ values.
3. Federating raw metrics. The global server’s head grows without bound; queries slow; memory pressure follows. The design anti-pattern, not a misconfiguration.
4. A leaf restart. /federate serves only what the leaf’s head has replayed; global graphs show minutes of gaps that look like a monitoring outage but are a scrape-source outage.
5. A network partition between leaf and global. After the lookback window, global panels for that leaf go blank — with no up of 0 for the leaf’s own targets, because the global server never scraped those targets. Only up of the federation job itself tells the truth.
6. Circular federation. Two servers each federating from the other re-ingest each other’s series with compounding staleness. It happens in organisations where two teams each believe they own the global tier.

How to troubleshoot it

1. up{job=~"federate-.*"} — is the chain alive at all?
2. Time the endpoint by hand: curl -g -w '%{time_total}\n' against the exact /federate URL, and compare with scrape_timeout. Slow selectors are the usual suspect.
3. Inspect labels on the global server. Are original job/instance preserved? Is the dc external label present? Collisions and missing external labels are visible immediately.
4. Look at the leaf. CPU at federation-scrape moments, and its own query latency. Reduce selector count or precompute more with recording rules.
5. Audit series volume on the global server with /api/v1/status/tsdb (seriesCountByMetricName) rather than a fleet-wide count query, which is itself expensive.

Security implications

/federate is an unauthenticated read of whatever the requester selects. Anyone who can reach port 9090 can ask for a broad match[] and walk away with your fleet’s metrics — hostnames, topology, capacity and worse. On an otherwise unprotected Prometheus it is a data-exfiltration path, not a health endpoint. Network-policy it to the global tier, or put the leaf behind an authenticating proxy and give the federation job credentials.

Performance implications

Each selector costs the leaf one instant query per scrape; few selectors over pre-aggregated series keep that cheap. A 60s federation interval is plenty for aggregates — the global view does not need 15s resolution. On the global server, storage cost is simply the selected series count at the federation interval, which is why curating the selector set is the entire capacity plan.

Verification

You should now be able to answer:

• What does /federate actually do when a scrape arrives?
• Why is honor_labels: true required on the federation job, and what breaks without it?
• Name three limits that make federation the wrong tool for global queries over raw data.
• What happens to global history when the global server is down for an hour — and why?
• Which two metrics describe the health and cost of a federation job?

Quiz