Observability HA — the high availability of the observability stack

Observability HA is the discipline of ensuring the observability stack is available. The Prometheus HA, the Grafana HA, the Loki HA, the Alertmanager HA are the components. This lesson walks the observability HA, the per-component pattern, and the production patterns.

The observability HA

The observability HA:

flowchart LR
    A[Application] --> B[Prometheus pair]
    B --> C[Thanos sidecar]
    C --> D[S3]
    D --> E[Thanos query]
    E --> F[Grafana]
    G[Prometheus] --> H[Alertmanager pair]
    H --> I[Slack]
    H --> J[PagerDuty]
    A --> K[Loki]
    K --> L[S3]
    K --> M[Grafana]

The observability HA is the production pattern.

The Prometheus HA

The Prometheus HA:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: my-prometheus
spec:
  replicas: 2
  thanos:
    baseImage: quay.io/thanos/thanos
    version: v0.32.x
    objectStorageConfig:
      name: thanos-objstore-config
      key: thanos.yaml

The Prometheus HA is via the pair + Thanos sidecar.

The Grafana HA

The Grafana HA:

spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: grafana
        env:
        - name: GF_DATABASE_TYPE
          value: postgres
        - name: GF_DATABASE_HOST
          value: postgres:5432

The Grafana HA is via multiple replicas + shared database.

The Loki HA

The Loki HA:

# Loki distributed mode
ingester:
  replicas: 3
distributor:
  replicas: 3
querier:
  replicas: 3

The Loki HA is the distributed mode.

The Alertmanager HA

The Alertmanager HA:

spec:
  replicas: 2

The Alertmanager HA is via the gossip protocol.

The OTel Collector HA

The OTel Collector HA:

spec:
  replicas: 2

The OTel Collector HA is via the deployments.

The Thanos HA

The Thanos HA:

flowchart LR
    A[Prometheus 1] --> B[Thanos sidecar 1]
    C[Prometheus 2] --> D[Thanos sidecar 2]
    B --> E[S3]
    D --> E
    E --> F[Thanos store]
    F --> G[Thanos query]
    G --> H[Grafana]

The Thanos HA is via the sidecars + store + query.

The cross-cluster HA

The cross-cluster HA:

flowchart LR
    A[Cluster 1] --> B[Thanos sidecar 1]
    C[Cluster 2] --> D[Thanos sidecar 2]
    B --> E[S3]
    D --> E
    E --> F[Thanos query]
    F --> G[Single Grafana]

The cross-cluster HA is via the Thanos query.

The production patterns

The production patterns:

flowchart LR
    A[Prometheus HA] --> B[Thanos sidecar]
    B --> C[S3]
    C --> D[Thanos query]
    D --> E[Grafana]
    E --> F[Dashboard]
    A --> G[Alertmanager HA]
    G --> H[Slack]
    G --> I[PagerDuty]

The pattern is the production discipline.

The cross-course references

• The Prometheus course covers the HA.
• The Grafana course covers the HA.
• The Loki course covers the HA.
• The Alertmanager course covers the HA.

Quiz

Production discipline

• Use the Prometheus HA. The pair + Thanos sidecar.
• Use the Grafana HA. Multiple replicas + shared database.
• Use the Loki HA. Distributed mode.
• Use the Alertmanager HA. The gossip protocol.
• Use the cross-cluster HA. Thanos sidecar + S3.
• Document the HA. The components, the patterns.

The observability HA is the production discipline. Operating it well is the per-component HA, the cross-cluster HA, and the production patterns.