Prometheus — the cluster metrics foundation

Prometheus is the cluster’s metrics foundation. The Prometheus Operator manages the Prometheus, Alertmanager, and Grafana lifecycles. The kube-prometheus-stack is the bundle. This lesson walks Prometheus, the Operator, the service discovery, and the Grafana integration.

Prometheus on Kubernetes

Prometheus on Kubernetes is the metrics foundation:

flowchart LR
    A[Cluster] --> B[Prometheus]
    B --> C[Thanos]
    C --> D[S3]
    B --> E[Alertmanager]
    E --> F[Slack]
    B --> G[Grafana]

The Prometheus scrapes the cluster’s metrics; the Alertmanager routes the alerts; the Grafana visualizes the metrics.

The Prometheus Operator

The Prometheus Operator manages the Prometheus lifecycle:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: my-prometheus
spec:
  replicas: 2
  serviceAccountName: prometheus
  serviceMonitorSelector:
    matchLabels:
      app: my-app
  ruleSelector:
    matchLabels:
      app: my-app
  resources:
    requests:
      cpu: 1
      memory: 2Gi

The Operator creates the Prometheus StatefulSet, the Service, the ConfigMap, and the PVCs.

The kube-prometheus-stack

The kube-prometheus-stack is the bundle:

helm install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace

The bundle includes:

• Prometheus.
• Alertmanager.
• Grafana.
• kube-state-metrics.
• node_exporter.
• The ServiceMonitors.
• The recording rules.
• The alerts.
• The Prometheus Operator.

The chart is the production deployment.

The service discovery

The service discovery is the Kubernetes API:

flowchart LR
    A[Prometheus] --> B[Kubernetes API]
    B --> C[Service endpoints]
    C --> D[Pod metrics]
    B --> E[Pod IP]
    E --> D

The Prometheus queries the Kubernetes API for the endpoints of the pods; the Prometheus scrapes the endpoints.

The scrape config

The scrape config:

scrape_configs:
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: (.+)
        target_label: __address__
        replacement: ${1}

The config uses the Kubernetes API to discover the pods.

The annotations

The annotations:

metadata:
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "8080"
    prometheus.io/path: "/metrics"

The annotations tell the Prometheus to scrape the pod.

The ServiceMonitor

The ServiceMonitor is the Operator-style discovery:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-app
  labels:
    app: my-app
spec:
  selector:
    matchLabels:
      app: my-app
  endpoints:
  - port: http

The ServiceMonitor is the canonical Prometheus discovery.

The Grafana integration

The Grafana integration:

flowchart LR
    A[Prometheus] --> B[Prometheus datasource]
    B --> C[Grafana]
    C --> D[Cluster dashboard]
    C --> E[Workload dashboard]

The Grafana connects to the Prometheus via the Prometheus datasource.

The cross-cluster scaling

The cross-cluster scaling:

flowchart LR
    A[Cluster 1] --> B[Prometheus 1]
    C[Cluster 2] --> D[Prometheus 2]
    B --> E[Thanos]
    D --> E
    E --> F[S3]

The cross-cluster scaling is via Thanos.

The long-term storage

The long-term storage:

apiVersion: monitoring.coreos.com/v1
kind: ThanosRuler
metadata:
  name: my-thanos
spec:
  ruleSelector:
    matchLabels:
      app: my-app
  storageSpec:
    ephemeral: {}

The ThanosRuler manages the recording rules; the ThanosSidecar uploads the data to S3.

The cross-course references

• The Observability course covers the metrics philosophy.
• The Grafana course covers the dashboards.
• The kube-prometheus-stack documentation covers the bundle.

Quiz

Production discipline

• Use the Prometheus Operator. The canonical deployment.
• Use the kube-prometheus-stack. The bundle.
• Use the ServiceMonitor. The declarative discovery.
• Configure the long-term storage. Thanos + S3.
• Integrate with Grafana. The dashboards.
• Document the deployment. The bundles, the ServiceMonitors.

The Prometheus on Kubernetes is the cluster’s metrics foundation. Operating it well is via the Operator, with the ServiceMonitor for discovery, and the long-term storage for retention.