Duplicate Scraping · Observability

A paging alert fires on node_cpu_seconds_total. The on-call engineer opens Grafana. Every CPU panel shows roughly double the expected load. Two Prometheus data sources are configured in the dashboard. Grafana sums them. The on-call is looking at sum(rate(node_cpu_seconds_total[5m])) from two servers that scrape the same targets and have nothing in their configuration to mark which series came from which instance.

Duplicate scraping is the most common silent misconfiguration of Prometheus HA. Two servers scrape the same endpoints, store the same series under slightly different label sets, and report roughly twice the truth to every downstream consumer. The symptom is hidden until someone plots the same metric from two data sources and forgets to deduplicate.

What it is

Duplicate scraping is the practice of running two or more Prometheus instances that scrape the same target set, where “the same” means identical targets selected from identical service-discovery sources, with no coordination between the instances. Each Prometheus ingests its own copy of every series. The series are functionally identical but carry distinct internal and external labels that distinguish their origin.

A duplicate-scrape pair is not, by itself, a problem. It is the default pattern for Prometheus HA: two stateless replicas, each with the full target set, each scraping the full target set, each shipping to the same long-term store. The problem appears only when downstream consumers add the copies back together without deduplication. Grafana summing two identical metrics is the classical failure.

The contrast is scrape sharding. Sharded replicas split the target set between them. Each replica scrapes a disjoint slice. There are no duplicates because no target is scraped twice. Sharding sacrifices per-replica completeness: if one replica dies, half the targets are not scraped until something else takes its slice.

Why a sysadmin cares

Duplicate scraping is the cheapest HA pattern to deploy. It is also the most expensive HA pattern to operate. The cost shows up in three places:

Storage. Every series is stored twice (or more) on the long-term store. The Thanos Store gateway or the Prometheus TSDB has to index the duplicates.
Scrape load on targets. A Kubernetes node exporter that serves 200 scrapes per minute from one Prometheus now serves 400. The exporter is cheap, but the application exporter is often not. The duplicate scrape load is doubled on every endpoint, every interval.
Query correctness. A panel that sums rate(...) across two duplicate data sources returns roughly double the true value. An alert that counts up == 0 and uses count(...) counts the same outage twice if the alert fires twice (once per replica), and an alert that counts count(up == 0) against the same query path with dedup disabled can silently miscount.

The most operationally costly failure mode is silent duplication. The dashboard looks fine because it shows the same number twice on every panel, and the doubling is treated as “expected load.” The actual cause is discovered when someone subtracts the service-level metric from the cluster-level metric and the arithmetic does not balance.

How it works

Two Prometheus servers, identical scrape_configs, identical target selection:

   +-------------------+         +-------------------+
   | Prometheus A      |         | Prometheus B      |
   | replica=a         |         | replica=b         |
   |                   |         |                   |
   | scrape_configs:   |         | scrape_configs:   |
   |   job: node       |         |   job: node       |
   +---------+---------+         +---------+---------+
             |                             |
             | scrape every 15s            | scrape every 15s
             v                             v
   +-------------------+         +-------------------+
   | node-exporter     |         | node-exporter     |
   | 10.0.1.10:9100    |         | 10.0.1.10:9100    |
   +---------+---------+         +---------+---------+
             |                             |
             | returns metrics             | returns metrics
             v                             v
   +-------------------+         +-------------------+
   | TSDB A            |         | TSDB B            |
   | (or remote_write)  |         | (or remote_write) |
   +---------+---------+         +---------+---------+
             |                             |
             +--------------+--------------+
                            |
                            v
                    +---------------+
                    | Thanos Querier|
                    | (dedup on    |
                    | replica)     |
                    +-------+-------+
                            |
                            v
                      Grafana / API

Each Prometheus stores the same series under its own replica label. The Thanos Querier (or the Prometheus query layer, when the --query.replica-label flag is set) deduplicates by picking the sample with the highest timestamp for each series. The storage cost is doubled. The scrape load on every endpoint is doubled. The query result is correct only if dedup is configured on the read path.

The mental model is “each Prometheus writes its own view; the query layer reconciles.” Reconciliation is the part most teams forget to configure.

Under the hood

A Prometheus instance stamps its global.external_labels on every series it produces (and on every alert it sends). For a duplicate-scrape pair, the external labels are the only thing distinguishing the two copies of the same series from each other. The convention is to add replica: <unique-id> to each instance so the labels differ. If the labels are equal, the two Prometheuses write series that are identical at the label level, and the storage layer (Prometheus TSDB, Cortex, Mimir) either rejects them as duplicate samples or accepts both and becomes ambiguous.

The scrape path is independent per Prometheus. Each instance opens its own TCP connection to each target. Each connection generates a separate series of up and scrape_duration_seconds samples. The up time series for a single target from one Prometheus is up\{instance="10.0.1.10:9100", job="node", replica="a"\} = 1. From the second Prometheus, it is up\{instance="10.0.1.10:9100", job="node", replica="b"\} = 1. Without dedup, a query for up == 0 returns one row per Prometheus per failed target.

The dedup mechanism on the read path keeps the most recent sample per series (the dedup function compares timestamps and keeps the higher one). It does not average. It does not pick the median. The implication is that an instance that is consistently behind on its scrapes wins the dedup tiebreak for any series whose samples are older than the other instance, and loses on every series whose samples are fresher. The dedup is not idempotent across time windows.

Storage on the long-term store depends on the backend. Thanos Store indexes by the external label set; two replicas with the same replica value produce two indistinguishable blocks and the indexer refuses to accept both, so the second replica’s upload fails with a hash collision. Mimir assigns a __replica__ label and dedups at ingest only if the ingester-side dedup is enabled. Cortex behaves the same way. The default everywhere is “reject duplicates” or “store both under distinct labels” depending on the configuration; the operator has to set the policy deliberately.

The query engine of Prometheus itself, since 2.6, honours a --query.replica-label flag. When set, the engine selects one sample per series across all values of that label. The flag applies to all queries served by that Prometheus, including recording-rule evaluations. A single-instance Prometheus serving dashboards with --query.replica-label=replica is the simplest form of dedup.

How to configure it

The minimum setup is two Prometheus instances with identical scrape config but distinct external labels, and a query layer that deduplicates.

/etc/prometheus/prometheus-a.yml:

global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    region: eu-west-1
    environment: production
    replica: a              # disambiguates this instance
    cluster: prom-ha-demo

scrape_configs:
  - job_name: node
    static_configs:
      - targets: ['10.0.1.10:9100', '10.0.1.11:9100']
  - job_name: app
    static_configs:
      - targets: ['10.0.2.20:8080']

/etc/prometheus/prometheus-b.yml: identical except the external replica value is b. The rest of the file is byte-for-byte the same. Do not introduce drift between the two configs beyond the replica label. Drift causes one Prometheus to scrape a target the other does not, and the dedup breaks.

Run the second instance with a different data directory and a different web port:

# SEVERITY: SERVICE-IMPACT
prometheus \
  --config.file=/etc/prometheus/prometheus-b.yml \
  --storage.tsdb.path=/var/lib/prometheus-b \
  --web.listen-address=0.0.0.0:9091

Validate the configuration before starting:

# SEVERITY: READ-ONLY
promtool check config /etc/prometheus/prometheus-a.yml
promtool check config /etc/prometheus/prometheus-b.yml

Each instance needs the replica label. If it is missing, the two instances produce identical label sets on every series, and the downstream store sees them as duplicates with no way to distinguish them.

The dedup at query time on a single-instance Prometheus:

# SEVERITY: SERVICE-IMPACT
prometheus \
  --config.file=/etc/prometheus/prometheus-a.yml \
  --query.replica-label=replica

The flag tells the query engine to pick one sample per series across all values of replica. Without the flag, the engine returns every sample and the consumer sees the double.

For Thanos Querier (which serves the unified view), the equivalent is the --query.replica-label flag on the querier:

# SEVERITY: SERVICE-IMPACT
thanos query \
  --http-address=0.0.0.0:10902 \
  --grpc-address=0.0.0.0:10901 \
  --query.replica-label=replica \
  --store=thanos-store-a:10901 \
  --store=thanos-store-b:10901

The replica label is the default. Override only if the production convention uses a different label name (for example, prometheus_replica for Grafana Cloud).

How to validate it

The first check is that the two instances are running with different replica labels:

# SEVERITY: READ-ONLY
curl -s http://prom-a:9090/api/v1/status/config | \
  jq '.data.yaml' | grep -E 'replica|external_labels'
curl -s http://prom-b:9091/api/v1/status/config | \
  jq '.data.yaml' | grep -E 'replica|external_labels'

Expected output (illustrative, trimmed):

replica: a
external_labels:
  region: eu-west-1
  replica: a
  ...
replica: b
external_labels:
  region: eu-west-1
  replica: b
  ...

The second check is that the series carry the replica label. Pick a metric that both instances scrape and inspect it directly:

# SEVERITY: READ-ONLY
curl -s 'http://prom-a:9090/api/v1/series?match[]=up{job="node"}' | \
  jq '.data[].labels'

Expected output (illustrative):

{ "job": "node", "instance": "10.0.1.10:9100", "replica": "a" }
{ "job": "node", "instance": "10.0.1.11:9100", "replica": "a" }

The same query against prom-b returns the same two series with replica: b. The label distinguishes the copies.

The third check is that the dedup query returns a single sample per series:

# SEVERITY: READ-ONLY
curl -s --data-urlencode 'query=up{job="node"}' \
  http://querier:10902/api/v1/query | \
  jq '.data.result[] | {metric, value}'

Expected output (illustrative):

{ "metric": {"job":"node","instance":"10.0.1.10:9100"}, "value": [1700000000, "1"] }
{ "metric": {"job":"node","instance":"10.0.1.11:9100"}, "value": [1700000000, "1"] }

Two rows, one per target, no replica label. If the label is still present, the dedup is misconfigured. If there are four rows, two Prometheuses are scraping the same targets but the query layer is not dedupping.

The fourth check is that alerts fired through the same pipeline also carry the replica label:

# SEVERITY: READ-ONLY
amtool alert query --alertmanager.url=http://am:9093 \
  | grep replica

If replica: a and replica: b appear on the same alert name, the Alertmanager cluster is receiving both copies. The fix is to add --cluster.replica-label=replica to Alertmanager so it treats alerts from both sources as the same alert (see lesson 02-external-labels).

How it can fail

Six failure modes appear repeatedly in production.

Duplicate scraping without dedup. Two Prometheuses scrape the same targets, neither the Thanos Querier nor the consumer is configured to deduplicate. Symptom: every metric in Grafana is roughly double the true value; alerts that aggregate fire on double the real rate. The fix is to set --query.replica-label=replica on the read path or to introduce a query-layer that dedups.
Drift between the two configs. Prometheus A picks up a new scrape job before Prometheus B does. Symptom: a target that should be scraped twice is scraped once during the drift window; dedup returns the right value but only one replica knows about the target. The fix is to source both configs from a single template (file_sd_configs, Consul, etc.) and to apply the change to both at once.
Reusing the same replica label. Two instances both use replica: a because the config was copy-pasted without changing the label. Symptom: the storage layer rejects the second instance as a duplicate; the second instance produces constant hash conflict errors during block upload; thanos_object_storage_hash_conflicts_total rises. The fix is to assign a unique replica value to each instance.
Scrape budget exceeded on the target. Two Prometheuses scrape the same application every 5 seconds; the application’s instrumentation is not designed for double the request rate. Symptom: scrape_duration_seconds rises on the second replica; the target starts to time out; up == 0 fires. The fix is either to reduce the scrape interval, to move one replica to scrape a different subset (sharding), or to decouple the application’s instrumentation cost from the scrape rate.
Alertmanager treats the two replicas as independent alerts. The Alertmanager cluster does not deduplicate on the replica label. Symptom: pages fire twice for the same condition. The fix is to set --cluster.replica-label=replica on every Alertmanager node.
The two Prometheuses disagree on the time. Clock skew between the two hosts makes the dedup “keep the highest timestamp” rule inconsistent. Symptom: at any given moment, one Prometheus is winning dedup on most series; a slow drift in clock skew shifts the winner series by series. The fix is NTP, with the same source on both hosts.

How to troubleshoot it

The diagnostic order when a metric looks roughly double the expected value:

Confirm the duplicate is real. Pick a counter that should be monotonically increasing (for example, node_network_transmit_bytes_total{device="eth0"}) and query it from each Prometheus directly.
Confirm the replica label exists. Query for count by (replica) (up) on each Prometheus. The result should match the number of targets, one row per replica.
Confirm the query layer is configured to dedup. Inspect the --query.replica-label flag on the Prometheus or Thanos Querier serving the dashboard.
Confirm the dashboard data source is the dedup query, not both raw Prometheuses. A dashboard that sums across both raw Prometheuses without dedup will double every metric.
Confirm the Alertmanager cluster is dedupping. Inspect amtool alert query for duplicate alert entries with different replica values.
Confirm the two configs are identical except for the replica label. Use diff -u on the rendered configs after template expansion.

What happens if two Prometheus servers scrape the same target and the read path has no dedup?

Every series is returned twice. A sum(rate(node_cpu_seconds_total[5m])) panel returns roughly double the actual CPU usage. An alert on http_requests_total rate counts each request twice and pages on the wrong threshold. Recording rules that aggregate across all instances over-count. The cost is silent: nothing fails, nothing logs an error, the metrics look slightly off.

Grafana shows the double clearly if the metric is a gauge with a known value (node_memory_MemAvailable_bytes on a known host). The doubling is hidden on counters with no upper bound, where the “correct” rate is rarely known exactly. The on-call engineer who notices the discrepancy has to drill into the dashboard configuration, find that two data sources are summed, and trace the data sources back to two Prometheuses that are not deduplicating.

The fix is to choose one of three patterns: dedup at the query layer (Thanos Querier with --query.replica-label), dedup at the single-instance Prometheus serving dashboards (--query.replica-label flag), or split the targets between the two replicas (scrape sharding) so the duplicates do not exist.

Security implications

Scrape traffic is HTTP. The Prometheus to target connection is authenticated or not, depending on the exporter. Duplicate scraping doubles the request rate to the exporter. The exporter must be able to handle the rate, and the rate-limiting on the target (if any) must be sized for the doubled volume.

External labels carry cluster identity. A misconfigured replica label that exposes the internal hostname of the Prometheus server is an information disclosure. The convention is to use a short opaque identifier (replica: a, replica: b) rather than the FQDN.

Performance implications

The performance cost is roughly 2x on every layer:

Scrape load on the target doubles.
CPU on the Prometheus host roughly doubles for the scrape and the WAL append.
Memory roughly doubles for the head block and the query engine.
Disk doubles for the local TSDB if both Prometheuses retain the same window.
Network to the remote write doubles.

The total cost depends on the scrape interval, the number of series, and the cardinality. A 50 000-series Prometheus at 15s scrape interval is a 100 000-series Prometheus in the duplicate configuration. The cost is paid twice on every layer of the stack.

Production guidance

Use scrape sharding by default. Two Prometheus instances with disjoint target sets give HA with no duplication.
Use duplicate scraping only when the cost of the second instance is acceptable and the operational risk of half a fleet going dark is unacceptable.
Always set a unique replica external label on each instance.
Always set --query.replica-label=replica on the read path serving dashboards and the API.
Always set --cluster.replica-label=replica on Alertmanager so it treats alerts from both replicas as one.
Source both Prometheus configs from a single template so drift between them is impossible.
Monitor the number of series per Prometheus. A sudden doubling on one Prometheus and zero on the other means a service discovery rule has migrated from one to the other.

Verification

You should now be able to answer:

What does a dashboard show when two Prometheuses scrape the same target and the read path does not deduplicate?
Which label distinguishes the two copies of the same series from a duplicate-scrape pair?
Where in the stack is dedup applied (scrape, write, query, storage)?
What is the difference between duplicate scraping and scrape sharding?
How does a misconfigured replica label surface in the storage layer?