Long-term storage — Thanos for Prometheus

Thanos is the long-term storage for Prometheus. The architecture is the sidecar + store + query + compactor + ruler. The Prometheus Operator integrates Thanos via the ThanosRuler CRD. This lesson walks the architecture, the deployment, the bucket, and the production patterns.

The Thanos architecture

The Thanos architecture:

flowchart LR
    A[Prometheus 1] --> B[Thanos sidecar 1]
    C[Prometheus 2] --> D[Thanos sidecar 2]
    B --> E[S3]
    D --> E
    E --> F[Thanos store]
    F --> G[Thanos query]
    F --> H[Thanos compactor]
    F --> I[Thanos ruler]

The Thanos architecture is the long-term storage.

The Thanos sidecar

The Thanos sidecar:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: my-prometheus
spec:
  replicas: 2
  thanos:
    baseImage: quay.io/thanos/thanos
    version: v0.32.x
    objectStorageConfig:
      name: thanos-objstore-config
      key: thanos.yaml

The sidecar is the upload agent.

The Thanos object storage config

The object storage config:

apiVersion: v1
kind: Secret
metadata:
  name: thanos-objstore-config
stringData:
  thanos.yaml: |
    type: S3
    config:
      bucket: my-thanos-bucket
      endpoint: s3.amazonaws.com
      access_key: <key>
      secret_key: <key>

The config is the S3 bucket.

The Thanos store

The Thanos store:

apiVersion: monitoring.coreos.com/v1
kind: ThanosStore
metadata:
  name: my-thanos-store
spec:
  version: v0.32.x
  objectStorageConfig:
    name: thanos-objstore-config
    key: thanos.yaml

The store is the S3 query layer.

The Thanos query

The Thanos query:

apiVersion: monitoring.coreos.com/v1
kind: ThanosQuery
metadata:
  name: my-thanos-query
spec:
  version: v0.32.x
  store:
    - my-thanos-store
  replicas: 2

The query is the federated query layer.

The Thanos compactor

The Thanos compactor:

apiVersion: monitoring.coreos.com/v1
kind: ThanosCompactor
metadata:
  name: my-thanos-compactor
spec:
  version: v0.32.x
  objectStorageConfig:
    name: thanos-objstore-config
    key: thanos.yaml
  retentionResolutionRaw: 30d
  retentionResolution5m: 90d
  retentionResolution1h: 1y

The compactor compacts the data.

The Thanos ruler

The Thanos ruler:

apiVersion: monitoring.coreos.com/v1
kind: ThanosRuler
metadata:
  name: my-thanos-ruler
spec:
  version: v0.32.x
  ruleSelector:
    matchLabels:
      app: my-app
  objectStorageConfig:
    name: thanos-objstore-config
    key: thanos.yaml

The ruler evaluates the rules against the S3 data.

The Grafana integration

The Grafana integration:

flowchart LR
    A[Grafana] --> B[Thanos query]
    B --> C[Thanos store]
    B --> D[Prometheus 1]
    B --> E[Prometheus 2]
    C --> F[S3]
    D --> G[Thanos sidecar 1]
    E --> H[Thanos sidecar 2]
    G --> F
    H --> F

The Grafana queries the Thanos query; the Thanos query federates the Prometheus and the S3 data.

The observability

The observability:

# Thanos store metrics
thanos_objstore_bucket_ops_total

# Prometheus metrics
prometheus_tsdb_head_series

# Thanos ruler metrics
thanos_rule_evaluation_duration_seconds

The metrics are the input for the alerts.

The cross-cluster scaling

The cross-cluster scaling:

flowchart LR
    A[Cluster 1] --> B[Thanos sidecar 1]
    C[Cluster 2] --> D[Thanos sidecar 2]
    B --> E[S3]
    D --> E
    E --> F[Thanos query]
    F --> G[Grafana]

The cross-cluster scaling is via the Thanos query.

The cost optimization

The cost optimization:

# Local Prometheus: 15 days
# Thanos S3: 1 year (or longer)
# Storage cost: $0.023/GB/month for S3 standard
# Storage cost: $0.004/GB/month for S3 Glacier

The cost optimization is the long-term retention.

Cross-course references

• The Prometheus Operator course covers the integration.
• The Observability course covers the metrics philosophy.
• The S3 / object storage course covers the bucket.

Quiz

Production discipline

• Use the Thanos sidecar. For the upload.
• Use the Thanos store. For the S3 query.
• Use the Thanos query. For the federated query.
• Use the Thanos compactor. For compaction.
• Configure the retention. Per resolution.
• Monitor the Thanos metrics. The bucket, the query.

The Thanos is the long-term storage for Prometheus. Operating it well is via the sidecar, the store, the query, and the compactor, with the retention configured for cost optimization.