Observability stack components — the toolbox

The observability stack has multiple components: the collectors, the storage, the query layer, the dashboards, and the alerting. Each component has a role. This lesson walks the components, the deployment patterns, and the production architecture.

The stack components

flowchart LR
    A[Cluster] --> B[Collectors]
    B --> C[Storage]
    C --> D[Query]
    D --> E[Dashboards]
    D --> F[Alerting]
    E --> G[Operator]
    F --> G

The stack has five components: collectors, storage, query, dashboards, alerting.

The collectors

The collectors gather the signals:

The collectors are the entry points of the stack.

The storage

The storage persists the signals:

The storage is the memory of the stack.

The query layer

The query layer processes the signals:

# Prometheus query
sum(rate(http_requests_total[5m])) by (status)

# LogQL query
{service="nginx"} |= "error"

# TraceQL query
trace_id = "abc123"

The query layer is the mind of the stack.

The dashboards

The dashboards visualize the signals:

flowchart LR
    A[Prometheus] --> B[Grafana]
    C[Loki] --> B
    D[Jaeger] --> B
    B --> E[Cluster dashboard]
    B --> F[Workload dashboard]
    B --> G[Service dashboard]

Grafana is the dashboard tool.

The alerting

The alerting sends notifications:

flowchart LR
    A[Prometheus] --> B[Alertmanager]
    C[Loki] --> B
    D[Jaeger] --> B
    B --> E[Slack]
    B --> F[PagerDuty]
    B --> G[Email]

Alertmanager is the alerting tool.

The production stack

The production stack:

flowchart LR
    A[Cluster] --> B[Prometheus]
    A --> C[Promtail]
    A --> D[OTel Collector]
    A --> E[event-exporter]
    B --> F[Thanos]
    C --> G[Loki]
    D --> H[Jaeger]
    E --> I[S3]
    F --> J[Grafana]
    G --> J
    H --> J
    B --> K[Alertmanager]
    K --> L[Slack]

The stack is the combination of the canonical tools.

The stack deployment

The stack deployment is via the kube-prometheus-stack:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring

The Helm chart deploys Prometheus, Alertmanager, and Grafana.

For Loki:

helm repo add grafana https://grafana.github.io/helm-charts
helm install loki grafana/loki-stack \
  --namespace monitoring

For Jaeger:

helm repo add jaegertracing https://jaegertracing.github.io/helm-charts
helm install jaeger jaegertracing/jaeger \
  --namespace monitoring

The deployments are per tool.

The HA patterns

The HA patterns:

flowchart LR
    A[Prometheus] --> B[Prometheus HA pair]
    B --> C[Thanos sidecar]
    C --> D[Thanos store]
    D --> E[S3]

The Prometheus HA is a pair of Prometheus instances. The Thanos sidecar uploads the data to S3 for long-term storage.

flowchart LR
    A[Loki] --> B[Loki distributor]
    A --> C[Loki ingester]
    A --> D[Loki querier]
    A --> E[S3]

The Loki HA is a distributed deployment.

The cross-course references

• The Prometheus course (Part LXXXVIII) covers the metrics.
• The Loki course (Part LXXXIX) covers the logs.
• The Jaeger course (Part XC) covers the traces.

Quiz

Production discipline

• Deploy the canonical stack. Prometheus, Loki, Jaeger, Grafana.
• Use Helm charts. The kube-prometheus-stack, the loki-stack.
• Configure Thanos for long-term storage. S3 as the store.
• Configure the HA patterns. Prometheus pairs, Loki distributed.
• Configure the alerting. Alertmanager routes to Slack, PagerDuty.
• Document the stack. The components, the deployment, the HA.

The observability stack is the cluster’s senses. Operating it well is deploying the canonical stack, configuring the HA, and integrating the components.