Prometheus Operator vs kube-prometheus-stack — the deployment choice

The Prometheus Operator and the kube-prometheus-stack are two deployment options. The Operator is the CRDs; the stack is the bundle. The choice is based on customization needs. This lesson walks the difference, the deployment scope, the upgrade path, and the production patterns.

The two options

flowchart LR
    A[Prometheus deployment] --> B[Prometheus Operator]
    A --> C[kube-prometheus-stack]
    B --> D[CRDs]
    C --> B
    C --> E[Dashboards]
    C --> F[Alerts]
    C --> G[ServiceMonitors]

The Operator is the foundation; the stack is the bundle.

The Prometheus Operator

The Prometheus Operator:

flowchart LR
    A[Prometheus CR] --> B[Operator]
    B --> C[Prometheus StatefulSet]
    B --> D[Service]
    B --> E[ConfigMap]
    B --> F[PVC]

The Operator watches the Prometheus CRs and creates the underlying Kubernetes resources.

The Operator CRDs

The CRDs:

- Prometheus: the Prometheus server
- Alertmanager: the Alertmanager
- ThanosRuler: the ThanosRuler
- ServiceMonitor: the scrape target
- PodMonitor: the pod-level scrape
- PrometheusRule: the alerting/recording rules
- Probe: the blackbox exporter

The CRDs are the Operator’s interface.

The kube-prometheus-stack

The kube-prometheus-stack:

flowchart LR
    A[Helm chart] --> B[Operator]
    A --> C[Prometheus]
    A --> D[Alertmanager]
    A --> E[Grafana]
    A --> F[kube-state-metrics]
    A --> G[node_exporter]
    A --> H[ServiceMonitors]
    A --> I[PrometheusRules]
    A --> J[Grafana dashboards]

The stack is the bundle.

The customization

The customization:

The stack is the opinionated deployment.

The deployment

The deployment:

# Operator only
helm install prometheus-operator prometheus-community/prometheus-operator \
  --namespace monitoring \
  --create-namespace

# kube-prometheus-stack
helm install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace

The Helm chart is the canonical deployment.

The upgrade path

The upgrade path:

# Operator upgrade
helm upgrade prometheus-operator prometheus-community/prometheus-operator \
  --namespace monitoring

# Stack upgrade
helm upgrade prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring

The upgrade is via the Helm chart.

The custom Prometheus

The custom Prometheus:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: my-prometheus
spec:
  replicas: 2
  serviceAccountName: prometheus
  serviceMonitorSelector:
    matchLabels:
      release: my-prometheus
  resources:
    requests:
      cpu: 1
      memory: 2Gi

The custom Prometheus is a CRD instance.

The custom ServiceMonitor

The custom ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-app
  labels:
    release: my-prometheus
spec:
  selector:
    matchLabels:
      app: my-app
  endpoints:
  - port: http

The custom ServiceMonitor is the Operator’s discovery.

The production patterns

The production patterns:

flowchart LR
    A[Cluster] --> B[Operator]
    B --> C[Prometheus]
    B --> D[Alertmanager]
    B --> E[Thanos]
    C --> F[Thanos]
    F --> G[S3]

The production pattern is the Operator + Thanos + S3.

The cross-course references

• The Observability course covers the metrics philosophy.
• The Grafana course covers the dashboards.
• The Helm course covers the chart installation.

Quiz

Production discipline

• Choose the kube-prometheus-stack for most clusters. The bundle.
• Use the Prometheus Operator only for custom deployments. When the stack is too opinionated.
• Configure the upgrade path. The Helm chart.
• Use the CRDs for the lifecycle. The Prometheus, Alertmanager, ServiceMonitor.
• Use Thanos for long-term storage. The S3 backend.
• Document the deployment. The chart values, the custom CRs.

The Prometheus deployment choice is the cluster’s metrics foundation. Operating it well is choosing the right deployment, the right upgrade path, and the right customizations.