Thanos and the Query Layer

Two Prometheuses scrape the same targets. Both ship their TSDB blocks to an S3 bucket via Thanos Sidecar. The Thanos Querier fans out queries across both Sidecar stores, and a panel that should show one CPU value per host shows two, stacked, with no clear indication that one is from replica a and the other is from replica b. The dashboards are confusing the on-call.

Thanos is the query layer that solves this. The Sidecar uploads blocks to object storage; the Querier fetches blocks from multiple stores; the Querier deduplicates series across duplicate-scrape pairs by external label. With the right configuration, the consumer sees one series per source, and the doubling disappears.

What it is

Thanos is a set of components that turn a set of Prometheus servers into a globally queryable, long-term-retention metrics platform. The components relevant to HA are:

• Sidecar. Runs alongside each Prometheus. Reads the Prometheus TSDB on disk, uploads new blocks to object storage (S3, GCS, Azure Blob, MinIO), and exposes the local TSDB as a gRPC store endpoint.
• Store. Reads blocks from object storage, indexes them in memory, and exposes them as a gRPC store endpoint.
• Querier. Fans out queries to one or more store endpoints (Sidecar and Store), applies replica-label deduplication, and returns the result to the consumer (Grafana, API client).
• Compactor. Reads blocks from object storage, deduplicates them at the block level (one block per replica per time range), and produces compacted blocks for long-term storage.
• Ruler. Evaluates recording and alerting rules against the same store endpoints, with the same replica-label deduplication.

For HA, the relevant components are Sidecar, Store, and Querier. The Sidecar is paired one-to-one with each Prometheus; the Store is shared across all Sidecars; the Querier is the consumer-facing component.

The contrast is Thanos Receive. Receive is the alternative write path: Prometheuses remote_write to a Receive gateway instead of writing locally and shipping blocks via Sidecar. Receive is appropriate for Prometheus servers that should not hold state locally (for example, in a Kubernetes deployment where pods are short-lived). For HA, the Sidecar pattern is simpler and is the focus of this lesson.

Why a sysadmin cares

Without a query-layer dedup, the duplicate-scrape pattern described in lesson 01 produces roughly double the metric values on every dashboard. The on-call cannot tell whether the doubling is a real signal (a service is exporting twice the expected traffic) or a configuration error (two Prometheuses are scraping the same target). The investigation has to start from “is the dashboard correct?” before “is the service correct?”

Thanos Querier with the right dedup flag eliminates the doubling. The consumer sees one series per source, regardless of how many Prometheuses scraped the target. The cost is one extra component to operate and a small amount of latency for the dedup step.

How it works

   +-------------------+         +-------------------+
   | Prometheus A      |         | Prometheus B      |
   | replica=a         |         | replica=b         |
   | + Thanos Sidecar  |         | + Thanos Sidecar  |
   +---------+---------+         +---------+---------+
             |                             |
        upload blocks                 upload blocks
             |                             |
             v                             v
   +------------------------------------------------+
   |                Object Storage                  |
   |  blocks from A:   fingerprint ...replica=a     |
   |  blocks from B:   fingerprint ...replica=b     |
   +-----------------------+------------------------+
                             |
                             v
                  +----------------------+
                  | Thanos Store         |
                  | (reads object store) |
                  +----------+-----------+
                             |
                             v
                  +----------------------+
                  | Thanos Querier       |
                  | --query.replica-label|
                  |     =replica         |
                  | dedup: keep latest   |
                  | sample per series    |
                  +----------+-----------+
                             |
                             v
                       Grafana / API
                       one series per
                       logical source

The Sidecar is colocated with the Prometheus. It watches the Prometheus’s data directory and uploads new blocks to object storage as Prometheus writes them. The Sidecar also exposes the local TSDB as a gRPC store endpoint so the Querier can read recent data without going to object storage.

The Querier is the consumer-facing component. It accepts PromQL queries from Grafana or API clients, fans out the query to every configured store endpoint (Sidecar A, Sidecar B, Store), and applies deduplication. The deduplication groups series by every label except the listed replica label, then within each group keeps the sample with the highest timestamp.

For a duplicate-scrape pair, both Sidecars return their copy of the series. The Querier keeps the one with the higher timestamp and discards the other. The consumer sees one series.

Under the hood

How to configure it

The Sidecar runs alongside Prometheus. The minimal Sidecar configuration:

# /etc/thanos/sidecar.yml
type: S3
config:
  bucket: thanos-prod-eu
  endpoint: s3.eu-west-1.amazonaws.com
  access_key: <aws-access-key>
  secret_key: <aws-secret-key>

Start the Sidecar:

# SEVERITY: SERVICE-IMPACT
thanos sidecar \
  --tsdb.path=/var/lib/prometheus \
  --prometheus.url=http://localhost:9090 \
  --objstore.config-file=/etc/thanos/sidecar.yml \
  --grpc-address=0.0.0.0:10901 \
  --http-address=0.0.0.0:10902

The Sidecar reads the Prometheus TSDB on disk and exposes it as a gRPC store on port 10901.

The Querier configuration is simpler. It only needs the list of store endpoints:

# /etc/thanos/querier.yml (empty file; configuration is via flags)

# SEVERITY: SERVICE-IMPACT
thanos query \
  --http-address=0.0.0.0:10902 \
  --grpc-address=0.0.0.0:10901 \
  --store=sidecar-a.prod.example.com:10901 \
  --store=sidecar-b.prod.example.com:10901 \
  --store=store-gateway.prod.example.com:10901 \
  --query.replica-label=replica

The --store flags list the gRPC endpoints of the Sidecars and the Store gateway. The --query.replica-label flag tells the Querier to dedup on the replica label.

For a Cortex deployment, the equivalent dedup is on the ingester:

# cortex ingester config
ingester:
  lifecycler:
    ring:
      kvstore:
        store: consul
      replicate_factor: 3
  ha_tracker:
    ha_tracker_timeout: 30s
    accept_ha_samples: true
    expected_ha_replicas: 2

The accept_ha_samples: true flag tells the ingester to accept samples from both replicas of an HA pair and dedup them in memory. The expected_ha_replicas: 2 flag declares the expected number of replicas; samples beyond that count are rejected.

How to validate it

The first check is that every Sidecar is reachable from the Querier:

# SEVERITY: READ-ONLY
curl -s http://querier:10902/api/v1/stores | jq

Expected output (illustrative):

[
  { "name": "sidecar-a", "address": "sidecar-a.prod.example.com:10901", "lastCheck": "..." },
  { "name": "sidecar-b", "address": "sidecar-b.prod.example.com:10901", "lastCheck": "..." },
  { "name": "store-01",   "address": "store-01.prod.example.com:10901", "lastCheck": "..." }
]

Three stores registered. The Querier will fan out queries to all three.

The second check is that the dedup flag is honoured. Pick a metric that both Prometheuses scrape:

# SEVERITY: READ-ONLY
curl -s --data-urlencode 'query=count by (instance) (up{job="node"})' \
  http://querier:10902/api/v1/query | jq '.data.result'

Expected output (illustrative):

[
  {"metric": {"instance": "10.0.1.10:9100"}, "value": [1700000000, "1"]},
  {"metric": {"instance": "10.0.1.11:9100"}, "value": [1700000000, "1"]}
]

Two rows, one per target. If the result is four rows (two per target, with replica=a and replica=b labels), the dedup is not active.

The third check is that the Sidecar is uploading blocks. Query the Sidecar’s metrics:

# SEVERITY: READ-ONLY
curl -s http://sidecar-a:10902/metrics | grep thanos_shipper

Expected output (illustrative):

thanos_shipper_uploads_total 432
thanos_shipper_upload_errors_total 0

A non-zero upload count and zero errors. The error counter rising means the Sidecar cannot write to object storage.

The fourth check is that the Compactor is keeping up. The Compactor produces compacted blocks; without it, the object store grows indefinitely with overlapping blocks per replica.

# SEVERITY: READ-ONLY
curl -s http://compactor:10902/metrics | grep thanos_compact

Expected output (illustrative):

thanos_compact_blocks_cleaned_total 14
thanos_compact_group_merges_total 7

A non-zero compaction count. If the compactor has been running for hours and the counters are still zero, the Compactor is not progressing.

How it can fail

Six failure modes appear repeatedly in production Thanos deployments.

1. Dedup flag points to the wrong label. The Querier is configured with --query.replica-label=prometheus instead of --query.replica-label=replica. Symptom: every series is dedupped to one Prometheus, not one per logical source. The fix is to set the flag to the label that varies per duplicate-scrape replica of the same Prometheus (typically replica).
2. Sidecar upload fails with hash conflict. Two Prometheuses share the same prometheus external label. Symptom: thanos_shipper_upload_errors_total rises with error="hash conflict". The fix is to set a unique prometheus external label per instance.
3. Querier cannot reach a Sidecar. The Sidecar host is unreachable (DNS, firewall, network policy). Symptom: the affected Prometheus’s data is missing from the unified view. The fix is to restore connectivity and to monitor the thanos_query_store_apis_found metric.
4. Compactor falls behind. The Compactor cannot keep up with the block upload rate. Symptom: the object store accumulates overlapping blocks; the indexer slows down; the Querier returns results more slowly. The fix is to scale the Compactor (more CPU, more memory, more parallelism).
5. The replica label is not set on the Prometheus. The Sidecar uploads blocks; the series carry no replica label; the Querier cannot dedup. Symptom: the doubled-series problem from lesson 01 returns. The fix is to add the replica external label to the Prometheus and to restart.
6. Cortex ingester is configured for HA but the accept_ha_samples flag is false. Symptom: the second replica’s samples are rejected with a not accepting HA samples error; one Prometheus’s data is missing. The fix is to enable accept_ha_samples: true and to declare the expected_ha_replicas.

How to troubleshoot it

The diagnostic order when a Thanos-backed dashboard shows duplicated series:

1. Confirm the Sidecar stores are registered with the Querier. curl /api/v1/stores on the Querier.
2. Confirm the dedup flag is set on the Querier. Inspect the process flags.
3. Confirm the replica label is present on the series. Query the Querier for count by (replica) (up).
4. Confirm the Sidecars are uploading blocks. Inspect the thanos_shipper_upload_errors_total metric.
5. Confirm the Compactor is running. Inspect the thanos_compact_blocks_cleaned_total metric.
6. Confirm the network path between every Sidecar and the Querier. nc -zv sidecar-a 10901 from the Querier host.

Security implications

Thanos components expose gRPC and HTTP endpoints. The gRPC port (10901) is for inter-component traffic; it should be firewalled to allow traffic only from the cluster’s node IPs. The HTTP port (10902) is for the API and metrics; it should require authentication for the API endpoints that allow mutations (store removal, config reload).

Object storage credentials are stored in the Sidecar configuration file. The file should be readable only by the Sidecar user, and the credentials should be sourced from a secret store (Vault, AWS Secrets Manager) rather than committed to the repository.

The block upload identifies the block by the prometheus external label. A misconfiguration that exposes internal cluster identifiers in the prometheus label is a small information disclosure. The convention is a short opaque identifier (prod-eu-a, prod-eu-b).

Performance implications

The Sidecar adds CPU and memory overhead to the Prometheus host. The Sidecar reads the local TSDB on every block upload (typically every two hours); the CPU cost is small. The memory cost is dominated by the index, which scales with the number of series. For a 10 million series Prometheus, the Sidecar may use 4-8 GiB of memory.

The Querier is memory-bound. The Querier holds the result set from every store in memory during the dedup step. For queries that return many series (a count by (job) (up) against a fleet), the result set can be large. The Querier’s default query timeout is 2 minutes; long-running queries should be sized accordingly.

The Compactor is CPU-bound. The Compactor reads every block from object storage, deduplicates the data per replica, and writes compacted blocks back. The work scales with the total data volume. A Compactor that falls behind produces an ever-growing set of overlapping blocks in object storage, which the indexer must process on every query.

Production guidance

• Use the replica external label convention. The default --query.replica-label=replica matches the convention; a non-default label name requires every operator to know the convention.
• Run at least one Store gateway per availability zone. The Store reads object storage on every query; the latency penalty for a single Store is too high for a production dashboard.
• Run a single Compactor per cluster. Multiple Compactors race on the same blocks and produce duplicate work. The Compactor uses a --compactor.concurrency flag to parallelise within a single process.
• Monitor thanos_query_store_apis_found and alert when the count drops below the expected number of stores.
• Monitor thanos_compact_blocks_cleaned_total and alert when the rate drops to zero for more than a few hours.
• For Cortex or Mimir deployments, set accept_ha_samples: true on the ingesters and declare the expected number of replicas.

Verification

You should now be able to answer:

• What are the three Thanos components relevant to Prometheus HA, and what is the role of each?
• Which flag tells Thanos Querier to deduplicate series across duplicate-scrape pairs, and what is the default value?
• How does Cortex’s accept_ha_samples flag differ from Thanos Querier’s --query.replica-label flag, and where does the dedup happen in each?
• What is the most common symptom of a misconfigured --query.replica-label flag, and how do you diagnose it?
• What is the role of the Thanos Compactor, and how do you know it is keeping up?

Quiz