The Split-Brain Problem

A network blip separates two Prometheus replicas for 90 seconds. Both replicas continue scraping. Both replicas continue remote_write to the same Thanos Receive gateway. When the network heals, both replicas believe they are the primary; both have written overlapping blocks to the same bucket. The Thanos Store sees two blocks for the same time range and refuses to dedup cleanly. The dashboards show discontinuities. The team cannot tell which replica is authoritative.

Split-brain is the failure shape where two instances both believe they are the primary and write overlapping data. In distributed systems, the discipline is fencing: a mechanism that prevents both replicas from writing simultaneously. Prometheus has no built-in leader election; the fencing has to come from the configuration.

What it is

Split-brain is the production name for a failure shape where two members of an HA pair both act as the active member. For Prometheus, the active member is “the one that scrapes and writes.” In a duplicate-scrape pair, both members are active by design; the discipline is that they are distinct actives, scraping disjoint target sets, writing to disjoint external labels.

The failure shape appears when the two replicas stop being distinct:

• Both replicas scrape the same target (the scrape config drifts).
• Both replicas write to the same long-term store with the same external labels (the prometheus label is identical).
• Both replicas send alerts to the same Alertmanager cluster with the same labels (the replica-label flag is missing).

Each of these is a configuration error that produces duplicate writes, duplicated series, or duplicated alerts. The on-call sees the symptom and traces it back to a configuration that allowed two replicas to act as one.

The contrast is fencing: a discipline that prevents both replicas from writing simultaneously or from writing the same data. The classic fencing pattern in distributed systems is a shared lock service (etcd, ZooKeeper, Consul) that grants a lease to one replica and revokes it on failure. Prometheus does not use a lock service; the fencing comes from the configuration.

Why a sysadmin cares

Split-brain is the failure shape that turns HA into a worse problem than the single point of failure it was meant to solve. A single Prometheus that dies leaves the team blind for a few minutes. Two Prometheuses in split-brain mode produce duplicate data that the team cannot reconcile without a manual cleanup.

Three production failures trace directly to split-brain in Prometheus:

• Block upload conflicts. Two Prometheuses with the same prometheus external label upload blocks to the same bucket. The bucket rejects the second upload with hash conflicts. The data is lost or duplicated, depending on the backend.
• Alertmanager double pages. Two Prometheuses send the same alert to Alertmanager. The cluster processes both and fires two notifications. The on-call is paged twice.
• Dashboard discontinuities. Two Prometheuses write overlapping time ranges. The query layer picks the higher timestamp per series. The values jump from one replica’s view to the other as the dedup flips. The graph shows discontinuities that are not present in either replica’s data alone.

How it works

The fencing patterns:

   Pattern 1: Scrape sharding (disjoint targets)
   +-------------+         +-------------+
   | Prom A      |         | Prom B      |
   | scrape: A   |         | scrape: B   |
   | set         |         | set         |
   +------+------+         +------+------+
          |                      |
          v                      v
      disjoint                disjoint
      targets                 targets
      no overlap = no split-brain

   Pattern 2: Active-passive with lease
   +-------------+         +-------------+
   | Prom A      |         | Prom B      |
   | state: active         | state: passive
   | scrape: all |         | scrape: none
   +------+------+         +------+------+
          |                      |
          |         lease        |
          +-----> etcd  <-------+
          |                      |
          v                      v
   only one holds the lease at a time

Pattern 1 is the canonical Prometheus fencing. The two replicas scrape disjoint target sets. There is no overlap, so there is no split-brain. The trade-off is that one replica’s failure leaves half the fleet unscraped briefly. For most production deployments, this is acceptable.

Pattern 2 is the active-passive pattern with a lease service. One replica scrapes and writes; the other is on standby. The active replica holds a lease in etcd (or Consul, or ZooKeeper); the passive replica polls the lease and takes over if the active replica fails to renew. The lease prevents both replicas from being active simultaneously.

Pattern 1 is simpler and is the right pattern for most deployments. Pattern 2 is appropriate when any scrape gap is unacceptable and the cost of duplicate-scrape is acceptable.

Under the hood

How to configure it

Pattern 1 (scrape sharding) is the canonical Prometheus fencing. The two Prometheuses have disjoint target sets:

# /etc/prometheus/prometheus-a.yml (shard 0)
global:
  external_labels:
    cluster: prod-eu
    replica: a
    shard: '0/2'

scrape_configs:
  - job_name: node
    static_configs:
      - targets: ['10.0.1.10:9100', '10.0.1.12:9100']

# /etc/prometheus/prometheus-b.yml (shard 1)
global:
  external_labels:
    cluster: prod-eu
    replica: b
    shard: '1/2'

scrape_configs:
  - job_name: node
    static_configs:
      - targets: ['10.0.1.11:9100', '10.0.1.13:9100']

The two replicas scrape different targets. The replica label disambiguates the two; the shard label records which shard of two each replica represents. There is no overlap; a target is scraped by exactly one replica.

For service-discovery-based sharding, the convention is to use a hash function on the target identity:

# /etc/prometheus/prometheus-a.yml
scrape_configs:
  - job_name: node
    consul_sd_configs:
      - server: 'consul.prod.example.com:8500'
        services: ['node-exporter']
    relabel_configs:
      - source_labels: [__meta_consul_node]
        modulus: 2
        target_label: __tmp_shard
        action: hashmod
      - source_labels: [__tmp_shard]
        regex: '0'
        action: keep

The hashmod action assigns a shard to each target based on a hash of the target identity. The replica only keeps targets whose hash mod 2 matches its shard number. The other replica keeps the targets whose hash mod 2 matches the opposite number.

Pattern 2 (active-passive with lease) requires an external service. A common implementation uses etcd:

# /etc/prometheus/prometheus-active.yml
global:
  external_labels:
    cluster: prod-eu
    replica: a

# Active replica: scrape the full target set
scrape_configs:
  - job_name: node
    static_configs:
      - targets: ['10.0.1.10:9100', '10.0.1.11:9100']

The passive replica has no scrape_configs. It monitors the lease via a separate process (typically a sidecar container) and starts scraping when the lease becomes available. The lease implementation is not part of Prometheus; it is a custom wrapper.

For Alertmanager, the fencing flag is cluster.replica-label on every node:

# /etc/alertmanager/alertmanager.yml
cluster:
  listen-address: ""
  replica-label: replica
  peers:
    - am-a.prod.example.com:9094
    - am-b.prod.example.com:9094

The flag tells the cluster to treat alerts with different replica values as the same alert. Without the flag, both Prometheuses’ alerts are treated as distinct, and double pages fire.

How to validate it

The first check is the target set. The two replicas should scrape disjoint targets:

# SEVERITY: READ-ONLY
curl -s 'http://prom-a:9090/api/v1/targets?state=active' \
  | jq '[.data.activeTargets[].labels.instance] | sort -u'
curl -s 'http://prom-b:9091/api/v1/targets?state=active' \
  | jq '[.data.activeTargets[].labels.instance] | sort -u'

Expected output (illustrative):

[ "10.0.1.10:9100", "10.0.1.12:9100" ]
[ "10.0.1.11:9100", "10.0.1.13:9100" ]

Two disjoint sets. Any overlap is a split-brain risk.

The second check is the series count. The two replicas should have roughly half the fleet’s series each:

# SEVERITY: READ-ONLY
curl -s http://prom-a:9090/api/v1/status/tsdb | jq .headStats.numSeries
curl -s http://prom-b:9091/api/v1/status/tsdb | jq .headStats.numSeries

Expected output (illustrative):

25117
24893

Two values, each roughly half the fleet. A divergence (one replica with 50 000 series, the other with 50) indicates that one replica is scraping the full set and the other is scraping almost nothing.

The third check is the prometheus external label. The two replicas must have different values:

# SEVERITY: READ-ONLY
curl -s http://prom-a:9090/api/v1/status/config | grep prometheus:
curl -s http://prom-b:9091/api/v1/status/config | grep prometheus:

Expected output (illustrative):

    prometheus: prod-eu-a
    prometheus: prod-eu-b

If the values match, the Thanos Store will reject one replica’s uploads with hash conflicts.

The fourth check is the alertmanager dedup. Trigger the same alert from both Prometheuses and verify one notification:

# SEVERITY: READ-ONLY
amtool alert query --alertmanager.url=http://am:9093 \
  | jq '[.[] | select(.labels.alertname=="TestAlert")] | length'

Expected output: 1. If the result is 2, the cluster.replica-label flag is missing.

How it can fail

Six failure modes appear repeatedly when the fencing discipline breaks.

1. Two replicas scrape overlapping targets. A service- discovery change causes both replicas to pick up the same target. Symptom: the series count on both replicas is roughly equal; the dedup query returns the right value but the doubled scrape load shows up on the target. The fix is to re-pin the sharding.
2. Two replicas have the same prometheus external label. The config was copy-pasted without changing the label. Symptom: hash conflicts on the Thanos Store; one replica’s uploads fail. The fix is to set a unique prometheus label per instance.
3. Alertmanager has no replica-label flag. The Prometheuses are deduplicated correctly at the query layer but the alerts are not. Symptom: double pages for every condition. The fix is to add cluster.replica-label: replica to every Alertmanager node.
4. The active-passive lease service is down. Both replicas cannot reach the lease service. Symptom: both replicas either stop scraping (neither holds the lease) or both scrape (both believe they hold the lease). The fix is to make the lease service highly available, or to fall back to scrape sharding when the lease is unavailable.
5. The two configs drift after a Consul change. One replica picks up a new service before the other. Symptom: the drifted service is scraped by one replica only; coverage is lost for the other replica until the config is reloaded. The fix is to source both configs from a single template.
6. The replica label is removed after a config reload. A hot-reload picks up a configuration file that drops the replica external label. Symptom: every series is now identical between replicas; the Thanos Store rejects the second replica’s uploads with hash conflicts. The fix is to restore the external label and to restart.

How to troubleshoot it

The diagnostic order when split-brain is suspected:

1. Confirm the target sets are disjoint. Compare the active targets on both replicas.
2. Confirm the prometheus external label is unique per replica. Inspect both configs.
3. Confirm the series count is roughly half the fleet per replica. Inspect the head stats.
4. Confirm the alertmanager replica-label flag is set on every node. Inspect the cluster block.
5. Confirm the Thanos Store is not reporting hash conflicts. Inspect the thanos_object_storage_hash_conflicts_total metric.
6. Confirm the long-term store is not showing doubled blocks. Inspect the bucket’s block listing for overlapping time ranges with identical external label sets.

Security implications

The fencing mechanism relies on the lease service or the hash ring. A misconfigured lease service that allows anyone to acquire the lease is a security vulnerability; an attacker could acquire the lease, redirect the active Prometheus’s writes, and disrupt the alert pipeline. The lease service must require authentication and must revoke leases on credential rotation.

The prometheus external label is visible in every series and every block uploaded to the long-term store. A label that exposes internal cluster identifiers (hostnames, IPs, environment names) is an information disclosure. The convention is a short opaque identifier.

The Alertmanager cluster gossip port (9094) is not authenticated. The port should be firewalled to allow traffic only from the cluster’s node IPs. Exposing 9094 to the public internet allows any host to join the cluster.

Performance implications

Scrape sharding halves the series count per replica. For a 10 million series fleet, each replica carries 5 million series. The memory cost per replica is roughly halved; the disk cost is roughly halved; the CPU cost is roughly halved. The total cost across both replicas is roughly equal to the single-Prometheus cost (no duplication).

The active-passive lease pattern halves the active work: one replica scrapes; the other is idle. The cost on the active replica is roughly equal to the single-Prometheus cost; the passive replica carries a small baseline (lease polling, config reload). The total cost is slightly higher than the single-Prometheus cost but lower than duplicate-scrape.

The Thanos Receive hash ring pattern distributes writes across replicas. Each replica carries a subset of tenants; the cost is roughly 1/N per replica where N is the number of replicas. The total cost is roughly equal to the single-Prometheus cost.

Production guidance

• Use scrape sharding as the default Prometheus fencing. Two Prometheuses with disjoint target sets give HA with no duplication and no split-brain risk.
• Source the two configs from a single template to prevent drift. The template parameterises the replica and prometheus labels and the shard assignment.
• Always set a unique prometheus external label per replica. The label is the Thanos block owner identifier; two replicas with the same label produce hash conflicts.
• Always set cluster.replica-label: replica on every Alertmanager node. The flag is the alert-layer dedup.
• For active-passive deployments, use a lease service that supports fencing tokens (etcd v3). Avoid simple TTL leases that can split-brain during network partitions.
• For Thanos Receive deployments, use the hash ring to assign tenants to replicas. The pattern is built into Thanos and prevents two replicas from writing the same tenant.
• Monitor the target set. Alert when the active targets on two replicas overlap.

Verification

You should now be able to answer:

• What is split-brain in a duplicate-scrape Prometheus HA pair, and what is the most common configuration that produces it?
• Which is the canonical Prometheus fencing pattern, and what is the trade-off against duplicate-scrape?
• Why does every Alertmanager node need the cluster.replica-label flag, and what happens if one node is missing it?
• What is the role of the prometheus external label in fencing block uploads to a Thanos Store?
• Which Prometheus metric indicates that two replicas have overlapping target sets?

Quiz