Thanos Overview

Two production Prometheus servers, each with a hot spare on a different host, all four scraping the same targets. Each instance has its own 30-day TSDB; a query for “all metrics for this service, last month” hits whichever Prometheus the dashboard knows about and returns partial data. None of the four stores will keep data past 30 days. Last year’s comparison is already gone.

Thanos is the missing layer: an opt-in set of components that turns a fleet of vanilla Prometheus servers into a horizontally scalable long-term store and a single global query endpoint. You do not replace Prometheus; you bolt Thanos onto the side.

What it is

Thanos is a CNCF-graduated project that adds long-term storage, deduplication, and global query to a fleet of unmodified Prometheus servers. The integration point is the Prometheus TSDB on disk: a Sidecar containerised alongside Prometheus uploads its persisted blocks to an object store, and a Store gateway serves those blocks to a Querier that fans out across many sources. Compactor downsamples and cleans; Ruler evaluates rules globally; Receiver accepts Prometheus’s remote_write for shops that do not want a Sidecar per host.

The project is Kubernetes-friendly but not Kubernetes-required; all components run as plain binaries against any object store that implements the objstore.Bucket interface (S3, GCS, Azure Blob, MinIO, Swift, Tencent COS, Alibaba OSS).

Why a sysadmin cares

Thanos is the lowest-friction way to add long-term metrics storage to an existing Prometheus deployment. The reasons it is worth knowing:

1. No re-instrumentation. The scrape topology does not change. Existing Prometheus servers stay where they are; the Sidecar runs next to each one and uploads blocks.
2. Multi-host query. A single Thanos Querier fronts N Prometheus servers, N Sidecars’ uploaded blocks, and any number of remote_write Receiver-ingested tenants. The PromQL surface is identical to a single Prometheus.
3. Cost-efficient retention. Object storage costs a fraction of NVMe. Retaining a year of metrics is plausible in a way that is not plausible on local disk.
4. HA deduplication. Two Prometheus scraping the same target with distinct external_labels write to the same bucket; the Querier deduplicates by the replica label at query time.
5. Drop-in path to Mimir. The Thanos Sidecar -> bucket -> Store path is the conceptual ancestor of Mimir’s write path. Knowing Thanos makes the Mimir data flow legible.

How it works

The components and their roles:

                       Thanos Querier (global query)
                                |
                                v
                    +-----------+-----------+
                    |                       |
              Thanos Store           Thanos Receiver
              (block gateway)        (remote_write ingest)
                    |                       |
                    v                       v
                 Object                Distributor
                 store                 (in-memory hash)
                 (S3 / GCS /                  |
                  MinIO / Azure)              v
                    ^                   Thanos Ingester
                    |                   (TSDB head, ring)
                    |                         |
                    |                         v
                    |                   Object store
                    |                         ^
                    |                         |
              Thanos Sidecar          Thanos Compactor
              (next to Prometheus)     (downsampling, retention)
                    |                         |
                    v                         |
              Prometheus TSDB                 |
              (local blocks)                  |
                    +-------------------------+

Five components worth knowing by name and job:

• Sidecar — runs alongside each Prometheus. On a block-cut event (every two hours), it uploads the new block to the bucket using the Thanos shipper format. Reads the local TSDB for recent-data queries.
• Store — stateless gateway that fronts the bucket. Serves block indexes and samples to the Querier; caches label and postings in an in-memory LRU.
• Querier — stateless Prometheus-compatible query API. Fans out to any number of Stores and Sidecars, deduplicates by replica label, merges results.
• Compactor — single-instance (do not run more than one) background worker that applies retention, repairs corrupted blocks, and produces 5-minute and 1-hour downsampled blocks for the older time ranges.
• Receiver + Ingester + Distributor — the optional path for shops that want a Thanos-native remote_write ingest instead of per-host Sidecars. The Distributor hashes the tenant onto Ingester replicas; Ingesters buffer the head in a local TSDB and flush to the bucket.

How to configure it

The Sidecar is the smallest unit and the entry point:

# thanos-sidecar.yaml
type: SIDECAR
http_address: 0.0.0.0:10902
grpc_address: 0.0.0.0:10901

# Object store: any objstore.Bucket implementation
objstore.config: |
  type: S3
  config:
    bucket: thanos-metrics-prod
    endpoint: s3.eu-west-1.amazonaws.com
    region: eu-west-1
    access_key: "${AWS_ACCESS_KEY_ID}"
    secret_key: "${AWS_SECRET_ACCESS_KEY}"

# Talk to its paired Prometheus
prometheus.url: http://localhost:9090

# What to upload and how
shipper.upload_compacted: true
shipper.no_lock: false          # enforce the per-block lock

The Store gateway is paired with the bucket and an index cache:

# thanos-store.yaml
type: STORE
http_address: 0.0.0.0:10902
grpc_address: 0.0.0.0:10901

objstore.config: |
  type: S3
  config:
    bucket: thanos-metrics-prod
    region: eu-west-1

# Index cache: the dominant cost knob. Memcached or in-process.
index_cache_config:
  type: IN-MEMORY
  config:
    max_size: "2GB"
    max_items: 500000

# Bucket scanner; controls how the Store enumerates blocks.
sync_interval: 3m
block_sync_concurrency: 20

The Querier fronts it all and dedups at query time:

# thanos-querier.yaml
type: QUERIER
http_address: 0.0.0.0:10902
grpc_address: 0.0.0.0:10901

# Discover Stores and Sidecars via DNS or a file.
query.replica_label: replica       # required for HA dedup
query.replica_label: receive       # for remote_write multi-tenant dedup
query.timeout: 2m
query.max_concurrent_select: 300

How to validate it

# Sidecar: blocks it has shipped are visible on the local store
# gateway endpoint it advertises.
curl -s thanos-sidecar:10902/api/v1/status | jq '.status'

# Blocks actually in the bucket (using thanos tools)
thanos tools bucket inspect \
  --objstore.config-file=bucket.yaml \
  | head -20

# Store gateway: it sees the blocks
curl -s thanos-store:10902/api/v1/blocks
# {"status":"success","data":{"blocks":[...ULIDs...],"uploaded":[...]}}

# Querier: a real query against last week's data
curl -sG thanos-querier:10902/api/v1/query \
  --data-urlencode 'query=count(up{job="prometheus"})' \
  --data-urlencode 'time=2026-08-06T12:00:00Z'

Five metrics that prove Thanos is healthy:

# Sidecar is shipping new blocks (climbs every ~2 hours)
rate(thanos_shipper_uploads_total[1h])

# Store gateway sees the bucket
thanos_store_blocks_last_loaded_timestamp_seconds

# Querier is fanning out correctly (sources queried per request)
thanos_query_metadata_apis_dns_lookups_total

# Compactor is alive (must be exactly 1 instance)
thanos_compactor_aborted_partial_uploads_total

# No sync errors between components
rate(thanos_objstore_bucket_operation_failures_total[5m])

How it can fail

1. Sidecar cannot reach the bucket. Symptom: Sidecar logs bucket upload: NoSuchBucket or AccessDenied; block uploads stall; the Store gateway sees a growing gap. The Prometheus that pairs the Sidecar is still scraping and answering recent-data queries, so the failure is invisible until someone asks for historical data.
2. Compactor not running. Symptom: the bucket fills with raw 1-minute blocks forever; old blocks never downsample or expire; storage cost climbs without bound. A Compactor crash is the same symptom with thanos_compactor_aborted_partial_ uploads_total non-zero.
3. Two Compactors racing. Symptom: intermittent “compaction aborted because another compaction is in progress” log lines; blocks appear and disappear; query-time “block not found” errors against historical ranges. The lock object is being held by both; the second one must be killed.
4. Store index cache too small. Symptom: queries against the bucket are slow at first call, fast at second call. The in-process cache is evicting label sets faster than the query workload generates them. Bump max_size or move to memcached.
5. Querier discovery stale. Symptom: a new Sidecar or Store is not seeing queries; thanos_query_metadata_apis_ dns_lookups_total flat. The DNS-based service discovery has stale records or low TTL; restart the Querier or fix the discovery backend.
6. HA replica labels missing. Symptom: every dashboard series is doubled. Two Prometheus writing to the same bucket without external_labels: { replica: A } / { replica: B } are seen as one source; the Querier deduplicates by timestamp and picks one, but storage and ingest are doubled. The query.replica_label: replica is mandatory in the Querier.

How to troubleshoot it

1. Are blocks in the bucket? thanos tools bucket ls. If not, the Sidecar pipeline is the failure; if yes, the Store gateway or Querier is.
2. Is the Sidecar healthy? curl sidecar:10902/-/ready and the Sidecar metrics. Compare thanos_shipper_uploads_total against prometheus_tsdb_head_series; the upload rate should track the block-cut cadence.
3. Is the Store seeing them? curl store:10902/api/v1/ blocks returns the list of block ULIDs the Store has loaded from the bucket. A mismatch with thanos tools bucket ls is the cache or scan interval.
4. Is the Querier fanning out? The thanos_query_metadata_apis_dns_lookups_total and thanos_query_store_apis_dns_lookups_total should both be non-zero and steady.
5. Are samples actually merged? Run a PromQL query that hits data only in the bucket (e.g. 60 days old) and compare to a query against the originating Prometheus if it still has it. The two answers must agree.
6. Logs. Grep for “shipper”, “sync”, “compactor”, “blocks”. Each component narrates its own failures.

Security implications

• Bucket credentials. Thanos needs s3:GetObject, s3:PutObject, s3:ListBucket, s3:DeleteObject (for Compactor retention). The credentials are long-lived by default; prefer IAM roles on the host (EC2 instance profile, GKE workload identity) or short-lived STS tokens. A leaked credential can read every block in the bucket.
• Bucket policy. Set a bucket policy that restricts s3:DeleteObject to the Compactor’s role. A leaked Sidecar credential should not be able to delete blocks.
• mTLS between components. Sidecar, Store, Querier, and Receiver all speak gRPC. Production deployments put mTLS in front of the gRPC ports; the Store and Querier endpoints expose the Prometheus HTTP API on a separate port, which should be on a private network.
• Multi-tenant isolation. On the Receiver / Ingester path, the X-Scope-OrgID header is the tenant boundary. On the Sidecar path, each Prometheus is effectively its own tenant (you control external_labels). Mixing the two without a strict naming convention causes cross-tenant queries to silently succeed.

Performance implications

Thanos is cheaper to operate than Mimir but not free. The component costs:

Sidecar            ~ 100-300 MB RAM, minimal CPU
                   (block upload is the dominant cost)

Store gateway      RAM dominated by index cache;
                   a 5 M-series fleet needs 4-16 GB RAM
                   + 2-8 cores for query serving

Querier            ~ 2-4 cores per 1 000 qps
                   Memory: ~ 1-2 GB per shard of in-flight query

Compactor          ~ 4-8 cores, 8-16 GB RAM
                   + significant I/O on the bucket
                   (only one; do not scale)

The dominant Store cost is the index cache. The dominant Querier cost is the merge across many sources. The dominant Compactor cost is the bucket I/O; running it in the same region as the bucket is a free lunch.

Verification

You should now be able to answer:

• What does each of Sidecar, Store, Querier, and Compactor actually do?
• How does a Prometheus -> Sidecar -> bucket -> Store -> Querier query resolve?
• Why is the Compactor a singleton, and what happens if a second one starts?
• What does query.replica_label: replica buy you, and what is the failure shape if it is missing?
• Where does the index cache live, and why is it the dominant Store cost?

Quiz