External Labels

A paging alert comes in. The on-call engineer inspects the Alertmanager UI and sees two alerts with the same name, the same labels, the same active time. One has replica=a. The other has replica=b. Both pages were sent. The Alertmanager cluster received both copies and treated them as two distinct incidents because the cluster does not know that a and b are the same logical source.

External labels are the Prometheus convention that disambiguates this case. They are a small set of name/value pairs that every Prometheus instance stamps on every series it produces and on every alert it sends. The convention is to set a replica label per instance. The downstream dedup logic, on the read path and in Alertmanager, picks one sample and one notification per logical source.

What it is

External labels are configured in the global.external_labels block of prometheus.yml. The block is a map of label names to values that are added to every time series at scrape time and propagated to every alert and every remote-write batch.

For a single-instance Prometheus, the block is often empty or carries cluster-wide identity (cluster, region, environment). For a duplicate-scrape pair, the block carries a unique replica value per instance. The block is also where operators add the prometheus external label used by Thanos Sidecar to identify the source Prometheus during block upload.

The contrast is internal labels. Internal labels are added by Prometheus to every series from the scrape itself: job, instance, and any labels the target’s metrics endpoint returns. Internal labels are the same across replicas. External labels are the only thing that distinguishes the two copies of the same series from each other.

Why a sysadmin cares

External labels are the boundary between “two Prometheus servers with overlapping data” and “two Prometheus servers with unambiguous data.” Without a unique external label per instance, the two servers produce identical label sets on every series, and the downstream storage cannot distinguish them. With a unique external label, the dedup logic on the read path and the dedup logic in Alertmanager can collapse the two copies into one logical source.

Three production failures trace directly to misconfigured external labels:

• Silent duplication. Two Prometheuses share the same external label set. The Thanos Store indexes blocks by their external label fingerprint; the second replica’s uploads fail with hash conflicts. The team sees constant hash conflicts log lines and zero data from the second replica.
• Double paging. The Alertmanager cluster does not know that replica=a and replica=b are the same logical source. Pages fire twice for every condition. The on-call wakes up twice per outage.
• Lost dedup. The query layer is configured with --query.replica-label=replica, but the series do not carry a replica label. The engine returns every sample. The doubling the team expected to disappear stays.

External labels are cheap to configure and expensive to omit.

How it works

The flow from scrape to alert:

   +-------------------+
   | Prometheus A      |
   | global:           |
   |   external_labels:|
   |     replica: a    |
   |     cluster: prod |
   +---------+---------+
             |
   scrape_targets
             |
             v
   +-------------------+
   | series carries:   |
   |   job=node        |
   |   instance=...    |
   |   replica=a       |  <-- stamped at scrape
   |   cluster=prod    |
   +---------+---------+
             |
   remote_write / Thanos Sidecar
             |
             v
   +-------------------+
   | store / querier   |
   | dedup: keep one   |
   | sample per        |
   | replica value     |
   +---------+---------+
             |
             v
   +-------------------+
   | Alertmanager      |
   | cluster dedup on  |
   | replica label     |
   +---------+---------+
             |
             v
       one page per
       logical source

External labels are added to the time series at scrape time. The label set is then propagated through every layer that preserves labels: remote write, Thanos upload, recording rules, alerting rules. The label is part of the series identity. Two series with the same metric name and different replica values are distinct series in the index, even if every other label is identical.

The dedup convention is: keep the sample with the highest timestamp. When two Prometheus servers scrape the same target, the one whose scrape succeeded most recently wins for that series at that instant. The other sample is discarded. The “winner” rotates as scrapes succeed or fail; the consumer sees a stable series whose value comes from whichever replica was last successful.

Under the hood

How to configure it

The minimum configuration for a duplicate-scrape pair:

# /etc/prometheus/prometheus-a.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    cluster: prod-eu
    region: eu-west-1
    replica: a              # this Prometheus is replica a
    prometheus: prod-eu-a   # Thanos Sidecar reads this

# /etc/prometheus/prometheus-b.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    cluster: prod-eu
    region: eu-west-1
    replica: b              # this Prometheus is replica b
    prometheus: prod-eu-b

Two configurations, identical except for replica and the prometheus label. The prometheus label is the Sidecar’s block identifier; the replica label is the dedup key.

Validate before starting either instance:

# SEVERITY: READ-ONLY
promtool check config /etc/prometheus/prometheus-a.yml
promtool check config /etc/prometheus/prometheus-b.yml

Start each instance with the dedup flag on the read path:

# SEVERITY: SERVICE-IMPACT
prometheus \
  --config.file=/etc/prometheus/prometheus-a.yml \
  --storage.tsdb.path=/var/lib/prometheus-a \
  --web.listen-address=0.0.0.0:9090 \
  --query.replica-label=replica

The same flag on the Thanos Querier that serves Grafana:

# SEVERITY: SERVICE-IMPACT
thanos query \
  --http-address=0.0.0.0:10902 \
  --grpc-address=0.0.0.0:10901 \
  --query.replica-label=replica \
  --store=thanos-store-a:10901 \
  --store=thanos-store-b:10901

For Alertmanager, the flag is on the cluster block, not the global block:

# /etc/alertmanager/alertmanager.yml
cluster:
  listen-address: ""
  replica-label: replica
  peers:
    - am-a:9094
    - am-b:9094
    - am-c:9094

The Alertmanager reads cluster.replica-label at startup; if omitted, the cluster does not dedup across replicas and double pages are inevitable for any alert that is fired by both Prometheuses.

How to validate it

The first check is the configuration itself. The replica value must be unique per instance:

# SEVERITY: READ-ONLY
curl -s http://prom-a:9090/api/v1/status/config | \
  jq '.data.yaml' | grep -E 'external_labels|replica'
curl -s http://prom-b:9091/api/v1/status/config | \
  jq '.data.yaml' | grep -E 'external_labels|replica'

Expected output (illustrative):

external_labels:
  cluster: prod-eu
  replica: a
external_labels:
  cluster: prod-eu
  replica: b

If the replica values match, the dedup will fail downstream and the block upload will produce hash conflicts.

The second check is that the scraped series carry the label. Pick a metric and inspect the labels directly:

# SEVERITY: READ-ONLY
curl -s 'http://prom-a:9090/api/v1/series?match[]=up{job="node"}' \
  | jq '.data[].labels.replica' | sort -u
curl -s 'http://prom-b:9091/api/v1/series?match[]=up{job="node"}' \
  | jq '.data[].labels.replica' | sort -u

Expected output (illustrative):

"a"
"b"

One value per query. If either query returns an empty list or the value is null, the external label was not stamped.

The third check is the dedup query on the read path. The query should return one row per target, with the replica label stripped:

# SEVERITY: READ-ONLY
curl -s --data-urlencode 'query=count by (instance) (up{job="node"})' \
  http://querier:10902/api/v1/query | jq '.data.result'

Expected output (illustrative):

[
  {"metric": {"instance": "10.0.1.10:9100"}, "value": [1700000000, "1"]},
  {"metric": {"instance": "10.0.1.11:9100"}, "value": [1700000000, "1"]}
]

If the result is [{"metric":{"instance":"10.0.1.10:9100","replica":"a"},"value":[...,"1"]}, {"metric":{"instance":"10.0.1.10:9100","replica":"b"},"value":[...,"1"]}], the dedup is not active. Either --query.replica-label is unset, or the series do not carry the replica label.

The fourth check is Alertmanager. A page from a duplicate-scrape pair should appear once, not twice:

# SEVERITY: READ-ONLY
amtool alert query --alertmanager.url=http://am:9093 \
  | jq '.[] | {labels, annotations}'

Inspect the labels of an active alert. If both replica=a and replica=b appear, the cluster is not dedupping. Add replica-label: replica to the Alertmanager cluster block.

How it can fail

Six failure modes appear repeatedly in production.

1. The replica label is identical on both instances. The config was copy-pasted without changing the label. Symptom: the second Prometheus’s block uploads fail with hash conflicts; the querier cannot distinguish the two copies. The fix is to assign a unique replica value per instance and to restart both.
2. The replica label was set late. The HA pair ran for a month with no external label. The replica label is added later. Symptom: the old series in long-term storage have no replica label; the new series do. The dedup query returns every series, and only the new ones are dedupped. The fix is to backfill the label via re-upload or to accept the gap.
3. The Alertmanager cluster does not have the flag. The Prometheus side has the external label; the Alertmanager cluster was not updated. Symptom: pages fire twice for every condition, once per replica. The fix is to add replica-label: replica to the Alertmanager cluster block and to reload every node.
4. The query layer flag points to the wrong label. The --query.replica-label flag is set to prometheus instead of replica. Symptom: the dedup keeps every sample because every series has a unique prometheus value (the Thanos Sidecar identifier is per-instance). The fix is to set the flag to the actual label that varies per replica of the same Prometheus (typically replica).
5. The replica label is added but the scrape config drops it. A relabel rule strips replica from the labels before storage. Symptom: every series looks identical again, dedup does nothing. The fix is to inspect the relabel rules and to ensure replica is preserved.
6. The two configs drift after a service-discovery change. One Prometheus picks up a new scrape job from Consul; the other does not. Symptom: dedup works per series but one Prometheus sees a target the other does not. The fix is to source both configs from the same template.

How to troubleshoot it

The diagnostic order when dedup is misbehaving:

1. Confirm the external label is set on each Prometheus. curl /api/v1/status/config and grep for the replica value.
2. Confirm the series carry the label. Query count by (replica) (up) on each Prometheus directly.
3. Confirm the query layer is configured to dedup. Inspect the --query.replica-label flag on the Prometheus or Querier.
4. Confirm the dashboard data source is the dedup query, not both raw Prometheuses.
5. Confirm the Alertmanager cluster is dedupping. Inspect amtool alert query for duplicate alert entries.
6. Confirm the relabel rules do not strip the replica label. Inspect the metric_relabel_configs on every scrape job.

Security implications

External labels are visible in every series, every recording rule, every alert, and every Grafana panel. A label that embeds internal hostnames, IPs, or environment identifiers is a small information disclosure. The convention is to use short opaque identifiers (replica: a, replica: b) for the replica label and stable identifiers for the cluster labels.

The Alertmanager cluster gossip replicates alert label sets across nodes. The cluster.replica-label is processed in memory; it does not appear in the alert payload sent to downstream receivers. The dedup happens before the integration call, so the receiver sees one alert per logical source.

Performance implications

External labels add one label to every series. The cost is one extra entry in the label index per series. For a Prometheus with 10 million series, the cost is one extra index entry per series, or roughly 10 million extra label-name lookups per query. The query engine groups series by every label except the replica flag, so the dedup path adds a constant-factor overhead on every query that uses the replica label.

The dedup itself is a one-step “keep highest timestamp” per series. The cost is O(N) where N is the number of series matching the query. The overhead is small compared to the query itself.

The Alertmanager dedup is also constant-factor. The cluster replicates every alert to every node regardless; the dedup filters before notification.

Production guidance

• Set the replica label on every Prometheus instance, even a single-instance deployment. The cost is one label. The benefit is that adding a second instance does not require a config rewrite on every downstream consumer.
• Use a short opaque identifier for the replica value (a, b, 01, 02). Avoid FQDNs or IPs that may change on host rebuild.
• Add prometheus: <unique-id> to external labels for Thanos Sidecar block ownership. The Sidecar uses this label as the block fingerprint.
• Always set --query.replica-label=replica on the read path serving dashboards and the API, even if the deployment is currently single-instance.
• Always set cluster.replica-label: replica on every Alertmanager node in the cluster.
• Source both Prometheus configs from a single template so drift between them is impossible. The labels replica and prometheus are the two values that must differ.
• After adding the external label to a long-running deployment, expect a discontinuity in the long-term store. The old series have the old label set; the new series have the new. Query deduplication will only apply to the new series. Backfill is rarely worth the complexity; the gap is usually accepted.

Verification

You should now be able to answer:

• Where in the Prometheus configuration is the external label defined, and which surfaces is it stamped onto?
• What is the conventional label name for distinguishing duplicate-scrape replicas?
• Which flag tells the Prometheus query engine to dedup on a label, and where does that flag go on Thanos Querier?
• Which flag tells Alertmanager to dedup across replicas, and why does every node in the cluster need it?
• What is the symptom of two Prometheuses with identical external label sets writing to the same Thanos Store?

Quiz